Recommender Forest for Efﬁcient Retrieval
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recommender systems (RS) have to select the top-n items from a massive item set.
For the sake of efﬁcient recommendation, RS usually represents users and itemsas latent embeddings and relies on approximate nearest neighbor search (ANNs)to retrieve the recommendation results.
Despite the reduction of running time, therepresentation learning is independent of ANNs index construction; thus, the twooperations can be incompatible, which results in a potential loss of recommendationaccuracy.
To overcome the above problem, we propose the Recommender Forest(a.k.a., RecForest), which jointly learns latent embedding and index for an efﬁcientand high-ﬁdelity recommendation.
RecForest consists of multiple K-ary trees, eachof which is a partition of the item set via hierarchical balanced clustering such thateach item is uniquely represented by a path from the root to a leaf.
Given sucha data structure, an encoder-decoder-based routing network is developed: it ﬁrstencodes the context, i.e., user information, into hidden states; then, leveraging atransformer-based decoder, it identiﬁes the top-n items via beam search.
Comparedwith the existing methods, RecForest brings in the following advantages: 1) thefalse partition of the near-boundary items can be effectively alleviated by the use ofmultiple trees; 2) the routing operation becomes much more accurate thanks to thepowerful transformer decoder; 3) the branch parameters are shared across differenttree levels, making the index to be extremely memory-efﬁcient.
The experimentalstudies are performed on six popular recommendation datasets: with a signiﬁcantlysimpliﬁed training cost, RecForest outperforms competitive baseline approaches interms of both recommendation accuracy and efﬁciency.
The code is available athttps://anonymous.4open.science/r/RecForest-CE76/ .
Introduction
Recommender system (RS) is an important way to address the information overload problem.
Atypical recommender system needs to select the top-n items for users from a massive-scale item set.
For the sake of efﬁcient recommendation, RS usually calls for the collaboration of representationlearning and Approximate Nearest Neighbour search (ANNs).
In the ﬁrst place, users and items arerepresented by embeddings in the same latent space; in the second place, the item embeddings areorganized with a speciﬁc ANNs index, like SCANN and HNSW, such that the top-n recommendationto the user can be efﬁciently accomplished.
Despite that the recommendation process is greatlyaccelerated from the above workﬂow, the recommendation quality will probably be restricted, giventhat the representation model is independently learned and can be incompatible with the ANNs index.
In recent years, many efforts have been devoted to alleviating the incompatibility between the representation model and the ANNs index, especially the effort on joint optimization of both components.
One representative class of works is the tree-based deep model (TDM) [23] and the joint optimizedtree-based model (JTM) [22] from Alibaba.
In both works, the item set is organized with a binarytree structure: each internal node acts like a cluster center, and each leaf node corresponds to a unique item.
On top of the tree structure, a preference model is learned to route from the root tothe leaf nodes for the top-n recommendation results.
These works achieve empirical gains over theconventional two-stage methods; besides, they preserve competitive retrieval efﬁciency as the timecost is logarithmic to the size of the item set.
However, the existing tree-based recommenders are still restricted in many aspects.
Firstly, the itemset is hierarchically partitioned; as a result, it is challenging to route to items located around partitionboundaries.
Secondly, a routing decision is made without consideration of the routing trajectory(i.e. , from the root to the direct ancestor of the current node), so the accuracy of beam search can belimited.
Thirdly, the tree-based index can be memory-consuming, given that the number of internalnodes is at the same magnitude as the leaf nodes (i.e., the number of the items).
Last but not least, theexisting methods call for joint adaptation of the representation model and tree index, so that the treestructure needs to be repetitively updated, resulting in a signiﬁcant cost for the training stage.
To overcome the limitations, we propose a novel framework, Recommender Forest (a.k.a., RecForest)for an efﬁcient and high-ﬁdelity recommendation.
RecForest is highlighted for the following features.
• RecForest consists of multiple K-ary trees, each of which is a partition of the item set based on
balanced hierarchical clustering.
With the construction, the retrieval of the near-boundary items
can be effectively improved, as evidenced in Section 3.6.1, since a near-boundary item missed in one tree can be retrieved back on another tree.
• RecForest leverages a transformer decoder for beam search.
On top of such a decoder, the routing trajectory, i.e., from the root node to the current node, can be jointly considered when the nextrouting decision is made. Compared with the previous methods which merely take account of the
current node, the beam search becomes much more accurate since the routing trajectory is fully
utilized, as analyzed in Section 3.4. • The tree parameters are shared across different tree levels; in other words, there are mere K vectors (corresponding to the K different branches) in each K-ary tree.
Thanks to parameter sharing, RecForest becomes much more memory-efﬁcient compared with the existing tree-based recommenders, as demonstrated in Table theoretically and in Table
empirically. • Given the above settings, RecForest becomes much less sensitive to the partition of the item set. As a result, without any tree update, RecForest can perform remarkably better than TDM and JTMwith repetitive tree update, as shown in Table 3. Therefore, RecForest can avoid the repetitiveadaption of the tree structure, saving a considerable portion of the training cost.
We perform comprehensive evaluations on six popular recommendation datasets.
According to ourexperiment results, RecForest notably outperforms the existing tree-based recommenders in termsof recommendation quality and efﬁciency.
Besides, we empirically verify that RecForest can beeffectively trained with much less time cost, indicating its strong usability in real-world scenarios.
Recommender Forest2.1 Preliminary
As mentioned above, RS usually represents user and item as latent embeddings, where the user’spreferences for the items are measured by the inner product of the embeddings [13, 1, 9].
As a result,the top-n recommendation is boiled down to the maximum inner product search (MIPS) problem.
Forthe sake of efﬁcient recommendation, various ANNs indexes are leveraged in practice, such as the treebased index [12, 8]; the hash-based index [15, 16, 5, 11], the quantization-based index [2, 19, 3] andthe graph-based index [10, 17, 21], etc.
These indexes have been well implemented by toolkits, like
FAISS, which greatly facilitates the deployment of recommendation systems in practice.
However,one limitation of the current ANN-based methods is that the index construction and the representationlearning are decoupled, which will probably introduce incompatibility between both modules.
To mitigate the above problem, recent works were proposed to jointly optimize the representationmodel and index, where two representative works are TDM [23] and JTM [22].
Both methods arebased on the tree structures, where the top-n recommendation can be efﬁciently accomplished by alayer-wise beam search.
For example, as shown in Figure 1, when the beam size is 2 , these modelswill score the four children of previously retrieved 2 nodes, from which the two largest children willbe selected. Such a process is iteratively performed until the ﬁnal top-2 leaf nodes are retrieved.
𝑖start Multi-Head Attention 01 1 user info 011+ 𝑢 item user
Encoder
Add & Norm
Masked Multi-Head Attention
Add & Norm
Feed Forward
Add & Norm
Linear & Softmax
[Figure]
𝑖𝑖𝑖𝑖𝑖𝑖000 001100 101 110𝑖𝑖 5 𝑖𝑖𝑖𝑖𝑖000 001 010 011 100 𝑖 𝑖 𝑖𝒚 𝑖 4 =𝒚 𝑖 4 =
𝒀 𝑖 4 =
[Figure]
2.2 Overview
The framework of RecForest is overviewed as follows.
First ofall, the item set is partitioned based on hierarchical balanced
Kmeans, so that a tree structure is generated.
The tree structurewill enable an item to be efﬁciently retrieved in O (log N ) timecomplexity ( N is the number of the items).
Knowing that theitems located close to the partition boundaries may get falselyassigned to a branch missing from the retrieval process, we propose to leverage multiple diversiﬁed trees, where the missingprobability can be largely reduced.
One item is correspondingto a leaf node in the tree, which can be represented by a pathstretching from the route.
Suppose that the K-ary tree is utilized, each item can be represented by a sequence of branch ID ( 0 , ..., K − 1 ), whose length is (cid:100) log K N (cid:101) .
For example, in Figure 2(b), the item i 4 is represented by “011” and “110” in thecorresponding binary trees, respectively.
Given the hierarchicalnumbering of the items, the recommendation turns out to be a sequence-to-sequence problem: basedon the encoded user information, paths to the most preferable items are progressively decoded viabeam search, from which the top-n recommendation is made.
The overall framework of RecForest isshown in Figure 2(a).
For ease of understanding, let’s ﬁrst summarize the notations used in this paper.
Let x u denotethe user information of u , which can be either a behavior sequence or a user feature vector.
Lety i ∈ { 0 , K − 1 } H be the item i ’s branch sequence of length H and Y i ∈ { 0 , K − 1 } T × H be aconcatenation of item i ’s branch sequences of T trees.
Formally, RecForst models the conditionalprobability P
( Y i | x u ) with the seq2seq architecture.
2.3 Construction of Tree and Forest
As mentioned, the item set is organized by a tree structure, where semantically close items shouldfall into the same branch as much as possible.
We will construct multiple trees so as to alleviate thefalse assignment of items around boundaries.
Detailed workﬂows are illustrated as follows.
2.3.1 Tree Construction
The tree is expected to be balanced for the sake of retrieval efﬁciency.
Thus, we leverage hierarchicalbalanced clustering, where each cluster is evenly divided into K child clusters until each childcluster merely contains one single item.
Besides, we propose to randomly sample K H − N items( H = (cid:100) log K N (cid:101) is the height of the tree) from the entire item set, such that the constructed tree willbe a complete K-ary tree.
We suggest the following two methods for the balanced partition of the item set.
Random :
For eachcluster, the included items are randomly partitioned into K equally-sized subsets.
Such a methodis extremely simple; yet, the semantic relationship between items is ignored.
Balanced Kmeans(short for Kmeans): Particularly, we ﬁrst pre-train item embeddings based on an arbitrary off-theshelf recommendation model.
In our experiment, we use the Deep Interest Network (DIN) [20] forsequential recommendation and the Bayesian Personalized Ranking (BPR) [13] for non-sequentialrecommendation, given their effectiveness and popularity in corresponding scenarios.
With such arecommendation model, the items’ semantic closeness can be measured by their embedding similarity,as relevant items will be close to each other in the latent space.
Then, for each cluster, the includeditems are evenly partitioned via Kmeans w.r.t.
their embedding similarity.
(Due to the space limitation,details about balanced Kmeans clustering are given in the supplemental material.)
2.3.2 Forest Construction
Different trees are desirable of being diversiﬁed such that the boundary items can be better coveredfrom beam search.
To this end, we propose a simple but effective approach, which combines therandom and Kmeans tree construction.
Particularly, for each cluster with n items, n mod K itemsare ﬁrst randomly chosen, from which one item is sampled with replacement into each child cluster;then, the remaining items are further partitioned into the K equally-sized groups via balanced Kmeans.
By doing so, different trees are naturally diversiﬁed due to the inherent randomness.
2.4 RecForest Enocoder
As shown in Figure 2(a), the encoder is utilized to encode user information and output user representation.
Any neural network based encoders are compatible with our framework.
We mainly considerthe sequential scenario and non-sequential scenario, where a sequence encoder and a feature encoderis used to encode user behavior sequences and feature vectors, respectively.
Feature Encoder
In the non-sequential scenarios, user information is denoted by a feature vector,i.e. x u .
We simply use a MLP for the feature encoder.
z u =
MLP ( x u ) .
When only user id isavailable, we can take its embedding as user representation, without any non-linear transformation.
Sequence Encoder
In the sequential scenarios, user information is an item sequence.
Concretely,x u =
[ i t 1 , i t 2 , . .
, i t m ] , where the item i t j ( 1 ≤ j ≤ m ) is the j -th interacted item of user u .
Itis straightforward to take item features into account.
In view of Transformer’s [18] state-of-theart performance in seq2seq tasks like machine translation and speech recognition, we utilize the
Transformer encoder for the sequence encoder.
In particular, z u = transformer-encoder ( x u ) , wherethe length of z u is the same as x u .
Due to space limitation, more details about the Transformerencoder are provided in the appendix.
2.5 RecForest Decoder
The decoder is to predict the next branch (routing decision) given the routing trajectory and theencoder’s outputs of user representation.
As shown in Figure 2(a), we apply the Transformer [18]decoder for this task due to its powerful capacity.
From down to up, following the branch embeddingand positional embedding, the Masked Multi-Head Attention [18] is utilized to model the complexdependence among branches at different layers.
Taking the outputs from the Masked Multi-Head Attention, Multi-Head Attention [18] is followed to encode the complex interactions between userrepresentation and routing trajectory representations.
The FFN layer then enhances non-linearity ofcontext-aware routing trajectory representations.
A linear layer with the softmax activation is appliedfor predicting the next routing decision.
Note that this decoder can also handle the non-sequencescenarios since a user feature vector can be considered as a sequence only with one element.
2.6 Training
As depicted in Figure 2(b), the routing trajectory of item i 4 is [0,1,1] and [1,1,0] on the two trees,respectively.
The framework uses [start,0,1] to predict [0,1,1] and uses [start,1,1] to predict [1,1,0].‘start’ is a special symbol for initializing the sequence, which actually corresponds to the treeroot.
According to experimental results, we do not share the transformer decoder among trees, i.e.,
P ( Y i | x u ) = (cid:81) k P θ k ( Y i [ k ] | x u ) , where θ k indicate the parameters of the k -th tree. In this case, alltrees share the same training procedure, so we illustrate the training with one tree.
At each layer, thetraining task is to predict the next routing decision given routing trajectory, which corresponds to amulti-class classiﬁcation problem.
Given a user u and its interacted item i , the loss for optimizingencoder and decoder is formulated as follows:
[Equation]
where y hi and y [0: h ] i indicate the branch (routing decision) at the h-th layer and routing trajectory untilthe h-th layer respectively, and ‘cat’ means concatenation.
2.7 Inference
Since each leaf in the tree corresponds to an item, RecForest transforms top-n recommendation into thesequence prediction problem.
In particular, we ﬁrst apply the encoder for deriving user representationand leverage the decoder to generate the top-n routing sequences, i.e., paths from the root node toleaf nodes on each tree, based on beam search.
Note that each item can correspond to multiple leafnodes (i.e. routing sequences) in each tree since some items are randomly sampled for building acomplete K-ary tree.
Therefore, the generated items should be duplicated.
Let C denote the union ofgenerated candidate items from the forest.
For c ∈ C , we ﬁrst compute the logarithmic probability 189 (cid:80) H −h =0 log Prob ( y hi
| cat ( start , y [0: h ] i ) , z u ) in each tree and then sum logarithmic probabilities over alltrees as the score of the item c .
We then select the top-n items from C for recommendation.
2.8 Complexity Analysis
Notation Denote by D item embedding size, B the beam search size, K the number of branches,and I the training set size.
Denote by T the number of trees in RecForest and the times of updatingthe tree in JTM and TDM.
In SCANN, denote by K vq the number of centroids in VQ, K pq thenumber of centroids in PQ, M the number of subspaces and W the number of probed VQ cells.
Time Complexity In inference, the beam search can be ﬁnished within O ( TKB log K N ) sincethe probabilities of KB branches at each level of all the T trees need to be calculated.
Hierarchical Kmeans clustering can be done within O ( NK log K N ) to build a tree index so that it takes
O ( TNK log K N ) to build the forest index.
Space Complexity
The memory consumption of the recommendation model can be regarded asthe constant consumption as it is inevitable in any recommender system so we mainly focus on thesize of the index structure.
In RecForest, we only need to keep the tree structure and a matrix withsize T × K × D , denoting branch embedding size.
In TDM and JTM, beside the tree structure, theyalso store all tree node embeddings, where the number of tree nodes is at the same magnitude as thenumber of the items.
The time complexity of IPNSW directly refer to HNSW, derived with exact Delaunay graphs.
[Table]
We summarize complexity analysis of typical algorithms in Table 1, which shows RecForest enjoys asmall index memory cost and low inference time.
This is because a smaller beamsize and fewer beamsearch are only required, and tree node representations are not stored but computed on the ﬂy.
Experiments
We conduct the experiments to answer the following research questions: RQ-1 : Does RecForestoutperform the SOTA efﬁcient recommenders in the tradeoff between efﬁciency and accuracy?
RQ2 : Does the forest-based index improve near-boundary item retrieval? RQ-3 : How much effectpositional embedding can take?
The experiments are carried out in both sequential scenarios andnon-sequential scenarios, where the sequence encoder and feature encoder is utilized, respectively.
Due to space limitations, we only report the results in the sequential scenario.
Other results areprovided in the Appendix.
These experiments are done on a Linux server with Tesla V100 GPUs.
3.1 Dataset
[Table]
We evaluate the RecForest with six real-worldrecommendation datasets, which can be downloaded from the url 1 .
The datasets are MovieLens 10M (abbreviated as Movie ), Amazon
Books (abbreviated as Amazon ), Tmall Click(abbreviated as Tmall ), Gowalla Check-in
Dataset (abbreviated as Gowalla ), Microsoft
News Dataset (abbreviated as MIND ).
Sincesome datasets only include rating-based explicitfeedback, they should be converted into implicitfeedback for RecForest’s inputs.
These datasetsare pre-processed by ﬁltering the users who interact with no more than 15 items.
The overallinformation of datasets is summarized in Table 2.
3.2 Baselines
In the paper, we report the results of the sequential recommendation scenario, so we compare theproposed RecForest with two-stage indexes, IPNSW [10] and SCANN [3], with learnable indexes,
TDM [23] and JTM [22] as well as with YoutubeDNN [1] and DIN [20] based on the brute-forceretrieval.
Note that brute-force-based YoutubeDNN and DIN are time-prohibitive in online services,but they can be considered as two strong baselines.
IPNSW and SCANN are built over itemembedding of a recommender.
The recommender shares the same encoder as RecForest, replaces thedecoder with ﬂat item embedding, and is trained with the same loss as DIN. Both TDM and JTMmake routing decision simply based on current node without the consideration of routing trajectories.
More details about baselines can be referred to the appendix.
3.3 Experimental Settings
In each dataset, we randomly choose 10% users as validation users, 10% users as test users, and allthe left users as training users.
Following TDM and JTM, we use a slide window to split user-iteminteraction histories into splices of length 70 at most.
For training users’ data, the ﬁrst 69 interactionsare used for input context and the 70-th item is regarded as the ground truth of prediction.
For data ofboth validation users and test users, we regard the ﬁrst half as context and others as ground truth.
The latent dimensionality is set to 96 in all methods.
Here, for a fair comparison of running time, weimplement SCANN and IPNSW with PyTorch.
To construct the quantization-based index, we follow SCANN’s default settings.
To construct the graph index of IPNSW, the maximum degree of eachnode is set to 16.
The beam size for any beam search is set to 100 unless speciﬁed.
To learn models,the learning rate is all set to 1e-3 with exponential decay.
The items’ representations for constructingtrees are obtained from item embedding of the pre-trained DIN on each dataset.
3.4 Comparison with Baselines
Settings We compare RecForest with the baselines on all datasets.
RecForest uses 2 trees on the
Movie and MIND, and 4 trees on other datasets.
Other settings can be referred to in Section 3.3.
[Table]
Results All the results are shown in Table 3, where the index memory cost of RecForest indicatesthe decoder’s memory consumption.
The following observations answers the RQ-1 .
• RecForest signiﬁcantly outperforms all efﬁcient recommenders with indexes on all datasetsw.r.t NDCG@20 and NDCG@40 .
The improvements over the best baselines w.r.t NDCG@20are 4.69%, 3.72%, 45.78%, 21.40%, 3.23%, 7.96% on the Movie, Amazon, Gowalla, Tmall,MIND, and Yelp dataset, respectively.
Its superiority over two-stage baselines (IPNSW andSCANN) demonstrates the effectiveness of joint representation learning, although the graph-basedindex shows a strong performance.
The higher accuracy than TDM and JTM indicates the beneﬁtof parameter sharing, routing trajectory utilization and the power of the transformer decoder.• RecForest performs better than brute-force-based YoutubeDNN on all datasets and eventhan brute-force-based DIN on the Movie, Gowalla, and MIND datasets.
Note that we donot apply DIN for post-reranking, otherwise the retrieval accuracy can be further improved.
This again conﬁrms the superpower of the transformer decoder.
Note that after ﬁne-tuning thehyper-parameters, YoutubeDNN performs well compared to TDM and JTM, which may not beconsistent with the results in TDM and JTM.• RecForest also enjoys the smallest index memory cost compared to other methods , whichcan be supported empirically and theoretically.
This is because we only embed the branchesand positions instead of all tree nodes in TDM and JTM.
RecForest even costs fewer memoriesfor index than the most lightweight PQ-based index – SCANN, which only stores the codes ofdata points and codebooks.
In spite of good retrieval accuracy, the graph-based index IPNSW ismemory-heavy due to the storage of graphs and data points.
Note that both DIN and YoutubeDNNare based on exhaustive search so they don’t consume memory for indexes.• RecForest is almost as fast as and sometimes even faster than SCANN for the top-n itemretrieval , which is considered as the SOTA index for MIPS in retrieval efﬁciency and accuracy.
This beneﬁts from transforming the top-n recommendation into sequence prediction, which canbe efﬁciently accomplished by beam search on GPU.
Note that the running time cost of IPNSWseems not consistent with Table 1, this is because only the approximate proximity graphs insteadof exact Delaunay graphs can be efﬁciently constructed for query in practice. Theoretical resultsshow that the RecForest is almost as efﬁcient as TDM and JTM, while the experimental resultsindicate RecForest is slightly faster. On the one hand, this is because we use the comparativelylarge branches (much larger than 2 used in TDM and JTM), remarkably decreasing the tree depthand thus reducing the times of beam search. On the other hand, because RecForest uses the simplefunction for routing decision and enables high parallelism due to the use of transformer decoder.
3.5 Extensive Study between Efﬁciency and AccuracySettings
The efﬁciency-accuracy curve is a commonly-used standard for evaluating the ANNsindex, so we also provide this result for better illustrating the superiority of RecForest.
The study ismainly investigated on the Amazon, MIND, Gowalla, and Yelp dataset, since the other two datasetsshow a similar trend.
To vary the retrieval time of RecForest, we adjust the beam size of beam searchfrom 10 to 100 with step 10, where the forest consists of at most 5 trees.
For TDM, JTM, and IPNSW,we adjust the beam size from 10 to 200 with step 10.
For SCANN, we adjust the number of probecells from 50 to 2,000 with step 100.0.04 0.02 0.00 0.02 0. [Table] RecForest
[Figure] [Figure] [Figure] [Figure]
Yelp
Results
The curves of different algorithms between NDCG@20 and query time are shown in Figure 3.
The following observations conﬁrms the answer of RQ-1 .
First, RecForest strikes thebest balance between query time and retrieval accuracy on all the four datasets , since the curvestands on top of the others.
This conﬁrms the superiority of RecForest to these competing baselines.
The advantage is the most signiﬁcant on the Gowalla dataset, evidenced by the biggest gap between RecForest and IPNSW. Second, With the increase of beam size, the accuracy of RecForest canimprove more signiﬁcantly than baselines.
This beneﬁts from the powerful transformer decoderand the novel training paradigm as well as the forest-based index, such that the representation modeland tree indexes are jointly learned better.
3.6 Ablation Study
3.6.1 Effect of Forest Construction
Settings We investigate three aforementioned ways of constructing a forest with at most 10 trees,where the branch number of each tree is set to 4.
The details about forest construction can be referredto Section 2.3.2.
Results
The results on the Amazon, Gowalla, MIND, and Yelp datasets are reported in Figure 4.
We can observe that Kmeans with randomness (i.e. Random+Kmeans) always performs bestand improves with the increasing number of trees while the Random performs worst and doesnot improve as much as Random+Kmeans when the tree number increases.
The reason why the
Random does not work well is that the semantic requirement is not satisﬁed.
Since Kmeans utilizesthe semantic information, it performs better than the Random, but is not affected a lot by growingthe number of trees.
This can be explained by the lack of diversity between trees.
This answers the RQ-2 : since the use of forest-based index can lead to signiﬁcant improvement of retrieval accuracy,particularly when tree construction takes semantics into account, the forest-based index improvesnear-boundary item retrieval .
Random
Kmeans
Random+Kmeans 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
N D C G @
Amazon 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
MIND 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
Gowalla 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
Yelp
3.6.2 Effect of Branch Number K
Settings We conduct evaluations on a single Kmeans-constructed tree on the Amazon, MIND,
Gowalla and Yelp dataset, where the branch number varies in {2,4,8,10,16,18}.
[Table]
Results The results are shown in Table 4.
We can observe that the retrieval accuracy of RecForest improves with thegrowing number of branches, particularly from 2 branches to 4.
The improvement on the Gowalla and Yelp datasets isthe most signiﬁcant.
This observation canbe explained as follows:
First, the vocabulary size grows, so the decoder can besupervised by more challenging discriminative signals and the decoders’ parameters grow.
Secondly, the routing sequenceswould be shorted, reducing the error accumulation of sequence prediction to some extent.
3.6.3 Effect of Positional Embedding
Settings Since the routing sequence is used for representing an item, the routing order should beimportant.
To this end, we compare four positional embeddings: (1) None indicates that there isno positional embedding; (2) Absolute positional embedding ( Abs for short) encodes the absolutepositions which range from 1 to maximum length of sequence by learnable parameters; (3) Relative
Key [14] ( RelK for short) focuses on attention and relative distance between tokens; (4) Relative Key
Query [6] ( RelKQ for short) is a reﬁned version of Relative Key.
Please refer to [6] for more details.
[Table]
Results
The results based on from single
Kmeans-constructed tree on the Amazon,
MIND, Gowalla, and Yelp datasets are summarized in Table 5.
We can observe that positional embedding takes a remarkableeffect on improving the retrieval capacity while different positional embeddingmethods do not have a signiﬁcant difference in retrieval accuracy , which answers the RQ-3 .
The effects of positionalembedding are more signiﬁcant on the Gowalla and Yelp datasets.
Conclusion and Future Work
In this paper, we propose the Recommender Forest for efﬁcient recommendation, which can be simplytrained within the sequence-to-sequence framework.
Even without updating the tree structure, RecForest enjoys a small index memory cost, low inference time, and highly-accurate recommendation.
The extensive study on six real-world recommendation datasets shows that RecForest becomes a stateof-the-art efﬁcient recommender.
In the future, we explore multi-task learning, non-autoregressiveprediction, and index structure learning as well as a more general framework.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please see the future work insection 4.(c)
Did you discuss any potential negative societal impacts of your work?
[No] For the
proposed RecForest model, we have not seen negative societal impacts(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide the URL in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see section 3.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendices for
the Submission: Recommender Forest for Efﬁcient Retrival
Our appendices include the following content:
• Sec.
A: the details about forest construction.
• Sec.
B: the illustration of Transformer encoder.
• Sec.
C: the pseudocode of training and inference.
• Sec.
D: the metrics used in performance evaluation.
• Sec.
E: the experiment details in sequential recommendation scenario.• Sec.
F: the experiments conducted in non-sequential scenarios.
A Forest Construction
A.1 Balanced Kmeans
For the sake of retrieval efﬁciency, the constructed tree is expected to be a complete K-ary tree.
However, the item number N does not necessarily meet this requirement.
We grow the dataset byinserting K H − N randomly sampled items from itself, such that it exactly contains K H items, where
H = (cid:100) log K N (cid:101) is the tree height.
We use the Balanced Kmeans in Algorithm 1 for hierarchicalitemset partition.
In particular, supposing a cluster contains K × m items, the algorithm splits theminto K clusters such that each cluster exactly contains m items.
At each iteration, we ﬁrst assigneach centroid the m closest unassigned items in order and then update the centroids according toitem assignment.
The clustering algorithm terminates until the assignments do not change any more.
Algorithm 1:
Data: Item set D , cluster number K
Result: { D 1 , D 2 , . . .
, D K } 1 m ← | D | K ; 2 Randomly initialize K centroids C
= { c 1 , c 2 , . .
. , c K } ; 3 repeat
U ← D ; for k = 1 → K do
U s ← Sort items in U ascendingly according to distance from c k ;
D k ← U s [0 : m − 1] ; c k ← 1 m (cid:80) x ∈ D k x ;
U ← U \ D k ; 10 until Assignments do not change ;
A.
2 Tree Construction
As described in Section 2.3.2, each tree is constructed by combining Random partition and Balanced
Kmeans to guarantee both diversity and semantic closeness of similar items.
Regarding the Randomstrategy, for each cluster, the included items are randomly partitioned into K equally-sized subsets.
When it is combined with balanced Kmeans, for each cluster with n items, n mod K items areﬁrst randomly picked, from which one item is sampled with replacement into each child cluster,and the remaining items are partitioned based on the Balanced Kmeans.
This method is shorten for
Random+Kmeans .
By applying the Random+Kmeans strategy recursively until each cluster onlycontains one item, the tree index can be constructed successfully.
Repeating the tree construction cangenerate diverse trees to form the forest.
We summarize the procedure in Algorithm 2.
B Transformer encoder
The Transformer encoder, shown in Figure 5, is to capture user’s interests from the sequential interactions x u =
[ i t 1 , i t 2 , . .
, i t m ] .
From down to up, following the interaction embedding and positional
Data: Dataset D , branch number K , random ratio δ
Result: Constructed tree 1 Q ← A queue with an element D ; 2 while Q is not empty
do
D top ← pop( Q ); n ← | D top | ;
D 0 ← Randomly draw n mod K items from D top without replacement ;
D top ← D top \ D 0 ; { D 1 , D 2 , . . .
, D K } ← Balanced-Kmeans( D top , K ); for k = 1 → K do i ← Sample one item with replacement from D 0 ; push( Q , { i } ∪ D k ); embedding, Multi-Head Attention [18] is utilized to model the interdependence of user’s each behavior.
Taking the output of Multi-Head Attention, the Feed Forward is to enhance the non-linearityof user’s behavior representation.
Finally, the hidden state, denoted as z u =
[ z 1 u ; z 2 u ; . . .
; z mu ] , isutilized to characterize user’s representation.
[Figure]
C Pseudocode of training and inference
In our RecForest, all the trees share the encoder but each tree has a unique decoder.
The trainingframework is summarized in Algorithm 3.
Note that we only give one training instance for clearillustration and it can be done in batch-wise manner for practical implementation.
In addition, theinference framework is summarized in Algorithm 4.
Once the generated candidates from each treeare collected, we compute the generated probabilities of candidates on each tree for ranking purpose.
Data: Forest = { tree 1 , tree 2 , . .
. , tree T } , training instance ( x u , i ), Encoder and Decoder t ( 1 ≤ t ≤ T ) w.r.t.
each tree 1 z u ← Encoder( x u ); 2 foreach t ∈ { 1 , 2 , . .
, T } do target ← get the path of item
i on the tree t ; decoder-input ← [‘start’,target[0:H-2]]; // H is tree height prediction ← Decoder t ( z u ,decoder-input);
L ( u, i ) ← loss(prediction, target) according to Eq (1); back-propagation from L ( u, i ) ; update Encoder and Decoder t by gradient descent;
Data: Forest = { tree 1 , tree 2 , . .
. , tree T } , Encoder and Decoder t ( 1 ≤ t ≤ T ) w.r.t.
each tree, user feature x u , beam size B , n (i.e. number of returning items)
Result:
The top- n items for user u 1 R ← {}; 2 z u ← Encoder( x u ); 3 for t ∈ { 1 , 2 , . . .
, T } do paths ← Beam-Search(Decoder t , z u , B ) ; foreach p ∈ paths do item i ← get-item( p, tree t );
R ← R ∪ { i } ; 8 foreach item i ∈ R do score [ i ] = 0 ; for t ∈ { 1 , 2 , . .
, T } do path ← get-path( i, tree t ); p t ( i ) ← compute the path probability based on Decoder t ; score [ i ] = score
[ i ] + log p t ( i ) ; 14 select the top- n items from R based on score ;
D Metrics
The ranking metric, Normalized Discounted Cumulative Gain( NDCG ), is used to measure theperformance of recommendation engine.
Discounted Cumulative Gain( DCG ) can be calculated by
[Equation]
log
[Equation]
1) where r i means the correlated score and k is the number of returned items.
As we focus on implicitfeedback, r i = 1 if the test user really interacts with the returned i -th item otherwise r i = 0 .
NDCGcan be calculated by
[Equation]
where iDCG means ideal DCG , i.e. ranking all the returned items according to correlated scores ina descending order and calculating the DCG on the ordered items.
Additionally, we also concern thememory consumption of retrieval index and inference efﬁciency.
Concretely, we record the size ofretrieval system and the time consuming in inference.
E Experiment Details in sequential recommendation scenarioE.1 Baselines in sequential recommendation scenario
• TDM [23] and its variant JTM [22], proposed by Alibaba Group, are the tree-based recommender
frameworks which train the user-item preference model and the tree structure simultaneously.
The
leaf nodes represent items and the internal nodes represent the clusters which the correspondingleaf nodes belong to.
Beam search is conducted on the tree to retrieve the items from coarse-grainclusters to ﬁne-grain clusters.
The main difference between TDM and JTM is the updating ofbijective mapping relation between items and leaf nodes.•
Deep Interest Network (i.e. DIN ) [20] utilizes the deep model to characterize the user’s preferences over items from user’s sequential interactions.
Concretely, DIN casts the training task asa binary-class classiﬁcation that reﬂects whether the user interacted with certain item.
As DINcan’t be used for efﬁcient recommendation, we report the brute-force-based performance.• YoutubeDNN [1], proposed by Google Group, is a deep neural network model which learns theuser representation and item representation from sequential user-item interaction history.
Then theinner product between user representation and item representation is used to capture user’s preference over the corresponding item, which converts the recommendation task to maximum innerproduct search(MIPS).
In our experiments, we mainly show the performance of YoutubeDNN byburte-force search for MIPS.• To solve MIPS efﬁciently, Quantizaiton-based method SCANN [3] and graph-based methodIPNSW [10] meet the start-of-the-art performance in each technique route respectively.
For faircomparison with our model, we model user’s interactions by Transformer encoder and replace thedecoder by item embedding.
The adapted model is trained in the same way as DIN to obtain theuser representation and item representation so that the learned representation can be used to buildthe SCANN index and IPNSW index.
The open-source code of TDM 2 , JTM 3 and DIN 4 are based on Alibaba Group’s X-DeepLeraning framework, we re-implement them by Pytorch.
And the open-source code of YoutubeDNN 5 is also available.
E.2 Detailed settings in
sequential recommendation scenario
In each dataset, we randomly choose 10% users as validation users, 10% users as test users and allthe left users as training users.
Following the settings in TDM [23] and JTM [22], we use a slidewindow to cut user-item interaction histories into splices where the window size is 70.
Each windowcontains at least 15 interactions and zero padding is applied if there is less than 70 interactions inthe window.
At training stage, the ﬁrst 69 user-item interactions are regarded as aware context andthe 70-th item is regarded as label in each window.
For validation users and test users, we regard thesecond half interactions as ground truths and the ﬁrst half interactions as observed interactions.
Thedimensionality of hidden space is set to be 96 for all methods.
The Deep Interest Network(DIN)
[20], which contains the embedding layer followed by a MLP withsize [128,64,2] and outputs the like dislike probabilities, is used as the the preference model for TDMand JTM.
YoutubeDNN [1] contains embedding layer followed by a MLP with size [128,64,96]and the number of negative sampling is set to be 1,000.
For SCANN, we follow the default settingsof the open sources 6 but implement them by Python and Pytorch for fair comparison in efﬁciency.
Concretely, the number of leaf is 2,000; the number of sub-space is 24 and the dimensionality of eachsub-space is 4; the number of code word in each sub-space is set to be 16; the value of threshold isset to be 0.2.
To construct the graph index of IPNSW, the maximum degree of each node is set to be16, the beam size is set to be 100.
For our RecForest, YoutubeDNN, TDM, JTM and DIN, Adam [7] is used as the optimizer andlearning rate is set to be 1.0e-3 with exponential decay.
In inference, the beamsize of RecForest,
TDM, JTM and IPNSW is
F Experiments in non-sequential scenarios
In non-sequential scenarios, we only apply user id to denote a user and the feature encoder to modeluser’s hidden representation.
Speciﬁcally, the learnable embedding vector of the user id denotesuser’s hidden representation.
F.1 Baselines
The classical deep model, Neural Collaborative Filtering ( NCF ) [4], characterizes user’s preferenceover item given the user id and item id. Due to the compatibility with any advanced user-item (oruser-node) preference model, we utilize NCF as the preference model in TDM [23] and JTM [22]frameworks.
For YoutubeDNN , we replace the user representation learning part used in sequentialrecommendation scenarios with the embedding vector of user id.
By this way, YoutubeDNN canbe compatible with non-sequential recommendation task.
SCANN and IPNSW are built on thelearned representation of YoutubeDNN.
Note that brute-force-based YoutubeDNN and NCF aretime-prohibitive in online services, but they can be considered as two strong baselines.
F.2 Experimental settings
In each dataset, we randomly choose 10% users as validation users, 10% users as test users and allthe left users as training users.
For validation users and test users, we randomly select a quarter of theinteracted items as ground truth, and put the rest into the training set.
The fusion NCF [4], whichcontains the GMF and MLP where the neuron numbers are set to be [512,256,128, 96] in each layer,is included.
Other necessary settings are as described in section E.2.
F.3 Comparison with Baselines
Settings We compare RecForest with the baselines on all datasets.
RecForest uses 2 trees on the
Movie and MIND, and 4 trees on other datasets.
Other settings can be referred to Section F.2.
Results All the results are shown in Table 6, where the index memory cost of RecForest indicatesthe decoder’s memory consumption.
From the table, we have the following
ﬁndings.
• Recforest nearly outperforms all efﬁcient recommenders with index on all dataset w.r.t.
NDCG@20 and NDCG@40 except the NDCG@40 on Movie where SCANN outperformsRecForest slightly.
Its superiority over two-stage baselines is not so signiﬁcant indicates thatinner product may be capable enough for representation learning to some extent in this simplerecommendation scenario.
The higher accuracy than TDM and JTM also verify the beneﬁt oflearning an indexing model with fewer parameters and the power of the transformer decoder.• RecForest performs better than brute-force-based NCF and YoutubeDNN on nearly alldatasets except the results on Tmall where YoutubeDNN outperforms RecForest slightly.
Thisphenomenon conﬁrms the the superpower of transformer decoder.• Being similar to the results of sequential recommendation scenarios, RecForest has the advantage in index memory cost , which can be highlighted by comparisons with baselines.
Theseobservations can be explained by referring to the same experiments in sequential scenarios.
Notethat NCF and YoutubeDNN are based on exhaustive search so they don’t consume memory forindexes.
F.4 Extensive Study between Efﬁciency and Accuracy
Settings Just like settings in sequential recommendation scenarios, we also draw the efﬁciencyaccuracy curve.
The study is mainly investigated on the Amazon, MIND, Gowalla, and Yelp dataset,since the other two datasets show a similar trend.
As described in the sequential scenario, we adjustthe beams size to vary the retrieval time cost for TDM, JTM, IPNSW and RecForest.
For SCANN,we adjust the number of probe cell to control the time cost.
Results
The curves of different algorithms between NDCG@20 and query time are shown in Figure6.
From the ﬁgure, we have the following ﬁndings.
[Table]
[Table]
[Figure]
• RecForest strikes the best balance between query time and retrieval accuracy on all the
four datasets , since the curve stands on top of the others.
This conﬁrms the superiority of
RecForest to these competing baselines.
The advantage is the most signiﬁcant on the MIND,
evidenced by the biggest gap between RecForest and IPNSW.
• With the increase of beam size, the accuracy of RecForest can improve more signiﬁcantly
than baselines.
This beneﬁts from the powerful transformer decoder and the novel training
paradigm as well as the forest-based index, such that the representation model and tree indexes
are jointly learned better.
F.5 Ablation Study
F.5.1 Effect of Forest Construction
Settings We investigate three aforementioned ways of constructing a forest with at most 10 trees,where the branch number of each tree is set to 4.
The details about forest construction can be referredto Section 2.3.2.
Results
The results on the Amazon, MIND, Gowalla and Yelp datasets are reported in Figure 7.
Theresults on Amazon, MIND and Yelp show the same trends as ones in sequential scenarios.
Concretely,
Kmeans with randomness (i.e. Random+Kmeans) always performs best and improves withthe increasing number of trees while the Random nearly performs worst and does not improveas much as Random+Kmeans when the tree number increases.
These results verify the ﬁndingsagain, i.e. the semantic information and the diversity among trees are important to improve therecommendation accuracy.0.04 0.02 0.00 0.02 0.
[Figure]
F.5.2 Effect of Branch Number K
Settings We conduct evaluations on a single Kmeans-constructed tree on the Amazon, MIND,
Gowalla and Yelp dataset, where the branch number varies in {2,4,8,10,16,18}.
[Table]
[Table]
[Table]
Results The results are shown in Table 7.
We can observe that the retrieval accuracy of RecForest on Amazon improveswith the growing number of branchesconstantly but shows no signiﬁcant inﬂuence on other three datasets.
Theseﬁndings indicate that the sensitivity ofbranch number depends on the dataset innon-sequential recommendation scenarios.
Generally, the increasing of branch numberand the reducing of path length don’t affectthe performance so much like they do insequential scenario.
F.5.3 Effect of Positional Embedding
Settings We also conduct four positional embedding methods like what we do in sequentialrecommendation scenario.
Concretely, (1) None ;(2) Abs ;(3) RelK ;(4) RelKQ and more details arereferred to section 3.6.3.
[Table]
Results
The results based on single
Kmeans-constructed tree on the Amazon,
MIND, Gowalla, and Yelp datasets are summarized in Table 8.
These results indicatesimilar ﬁndings as ones in section 3.6.3 thatpositional embedding takes a remarkable effect on improving the retrieval capacity while different positional embedding methods do not have a signiﬁcantdifference in retrieval accuracy .
Speciﬁcally, the inﬂuence of Abs and RelK is more signiﬁcant on Gowalla.