Recommender Forest for Efficient Retrieval
Anonymous Author(s)
AffiliationAddress email
Abstract
Recommender systems (RS) have to select the top-n items from a massive item set.
For the sake of efficient recommendation, RS usually represents users and itemsas latent embeddings and relies on approximate nearest neighbor search (ANNs)to retrieve the recommendation results.
Despite the reduction of running time, therepresentation learning is independent of ANNs index construction; thus, the twooperations can be incompatible, which results in a potential loss of recommendationaccuracy.
To overcome the above problem, we propose the Recommender Forest(a.k.a., RecForest), which jointly learns latent embedding and index for an efficientand high-fidelity recommendation.
RecForest consists of multiple K-ary trees, eachof which is a partition of the item set via hierarchical balanced clustering such thateach item is uniquely represented by a path from the root to a leaf.
Given sucha data structure, an encoder-decoder-based routing network is developed: it firstencodes user information into user representation; then, leveraging a transformerbased decoder, it identifies the top-n items via beam search.
Compared withthe existing methods, RecForest brings in the following advantages: 1) the falsepartition of the near-boundary items can be effectively alleviated by the use ofmultiple trees; 2) the routing operation becomes much more accurate thanks to thepowerful transformer decoder; 3) the branch parameters are shared across differenttree levels, making the index to be extremely memory-efficient.
The experimentalstudies are performed on six popular recommendation datasets: with a significantlysimplified training cost, RecForest outperforms competitive baseline approaches interms of both recommendation accuracy and efficiency.
The code is available athttps://anonymous.4open.science/r/RecForest-CE76/ .
Introduction
Recommender system (RS) is an important way to address the information overload problem.
Atypical recommender system needs to select the top-n items for users from a massive-scale item set.
For the sake of efficient recommendation, RS usually calls for the collaboration of representationlearning and Approximate Nearest Neighbour search (ANNs).
In the first place, users and items arerepresented by embeddings in the same latent space; in the second place, the item embeddings areorganized with a specific ANNs index, like SCANN and HNSW, such that the top-n recommendationto the user can be efficiently accomplished.
Despite that the recommendation process is greatlyaccelerated from the above workflow, the recommendation quality will probably be restricted, giventhat the representation model is independently learned and can be incompatible with the ANNs index.
In recent years, many efforts have been devoted to alleviating the incompatibility between the representation model and the ANNs index, especially the effort on joint optimization of both components.
One representative class of works is the tree-based deep model (TDM) [23] and the joint optimizedtree-based model (JTM) [22] from Alibaba.
In both works, the item set is organized with a binarytree structure: each internal node acts like a cluster center, and each leaf node corresponds to a unique item.
On top of the tree structure, a preference model is learned to route from the root tothe leaf nodes for the top-n recommendation results.
These works achieve empirical gains over theconventional two-stage methods; besides, they preserve competitive retrieval efficiency as the timecost is logarithmic to the size of the item set.
However, the existing tree-based recommenders are still restricted in many aspects.
Firstly, the itemset is hierarchically partitioned; as a result, it is challenging to route to items located around partitionboundaries.
Secondly, a routing decision is made without consideration of the routing trajectory(i.e.
, from the root to the direct ancestor of the current node), so the accuracy of beam search can belimited.
Thirdly, the tree-based index can be memory-consuming, given that the number of internalnodes is at the same magnitude as the leaf nodes (i.e., the number of the items).
Last but not least, theexisting methods call for joint adaptation of the representation model and tree index, so that the treestructure needs to be repetitively updated, resulting in a significant cost for the training stage.
To overcome the limitations, we propose a novel framework, Recommender Forest (a.k.a., RecForest)for an efficient and high-fidelity recommendation.
RecForest is highlighted for the following features.
• RecForest consists of multiple K-ary trees, each of which is a partition of the item set based on
balanced hierarchical clustering.
With the construction, the retrieval of the near-boundary items
can be effectively improved, as evidenced in Section 3.6.1, since a near-boundary item missed in
one tree can be retrieved back on another tree.
• RecForest leverages a transformer decoder for beam search.
On top of such a decoder, the routingtrajectory, i.e., from the root node to the current node, can be jointly considered when the next routing decision is made.
Compared with the previous methods which merely take account of the
current node, the beam search becomes much more accurate since the routing trajectory is fully
utilized, as analyzed in Section 3.4.
• The tree parameters are shared across different tree levels; in other words, there are mere K
vectors (corresponding to the K different branches) in each K-ary tree.
Thanks to parameter
sharing, RecForest becomes much more memory-efficient compared with the existing tree-based
recommenders, as demonstrated in Table 1 theoretically and in Table 3 empirically.
• Given the above settings, RecForest becomes much less sensitive to the partition of the item set.
As a result, without any tree update, RecForest can perform remarkably better than TDM and JTMwith repetitive tree update, as shown in Table 3.
Therefore, RecForest can avoid the repetitiveadaption of the tree structure, saving a considerable portion of the training cost.
We perform comprehensive evaluations on six popular recommendation datasets.
According to ourexperiment results, RecForest notably outperforms the existing tree-based recommenders in termsof recommendation quality and efficiency.
Besides, we empirically verify that RecForest can beeffectively trained with much less time cost, indicating its strong usability in real-world scenarios.
Recommender Forest2.1 Preliminary
As mentioned above, RS usually represents user and item aslatent embeddings, where the user’s preferences for the itemsare measured by the inner product of the embeddings [13, 1, 9].
As a result, the top-n recommendation is boiled down to themaximum inner product search (MIPS) problem.
For the sake ofefficient recommendation, various ANNs indexes are leveragedin practice, such as the tree-based index [12, 8]; the hash-basedindex [15, 16, 5, 11], the quantization-based index [2, 19, 3] andthe graph-based index [10, 17, 21], etc.
These indexes have beenwell implemented by toolkits, like FAISS, which greatly facilitates the deployment of recommendation systems in practice.
However, one limitation of the current ANN-based methodsis that the index construction and the representation learningare decoupled, which will probably introduce incompatibilitybetween both modules.
𝑖start
Multi-Head Attention 01 1 user info 011+ 𝑢 item user
Encoder
Add & Norm
Masked Multi-Head Attention
Add & Norm
Feed Forward
Add & Norm
Linear & Softmax
[Figure]
𝑖𝑖𝑖𝑖𝑖𝑖000 001100 101 110𝑖𝑖 5 𝑖𝑖𝑖𝑖𝑖000 001 010 011 100 𝑖 𝑖 𝑖𝒚 𝑖 4 =𝒚 𝑖 4 =
𝒀 𝑖 4 =
[Figure]
To mitigate the above problem, recent works were proposed to jointly optimize the representationmodel and index, where two representative works are TDM [23] and JTM [22].
Both methods arebased on the tree structures, where the top-n recommendation can be efficiently accomplished by alayer-wise beam search.
For example, as shown in Figure 1, when the beam size is 2 , these modelswill score the four children of previously retrieved 2 nodes, from which the two largest children willbe selected.
Such a process is iteratively performed until the final top-2 leaf nodes are retrieved.
2.2 Overview
The framework of RecForest is overviewed as follows.
First of all, the item set is partitioned based onhierarchical balanced Kmeans, so that a tree structure is generated.
The tree structure will enable anitem to be efficiently retrieved in O (log N ) time complexity ( N is the number of the items).
Knowingthat the items located close to the partition boundaries may get falsely assigned to a branch missingfrom the retrieval process, we propose to leverage multiple diversified trees, where the missingprobability can be largely reduced.
One item is corresponding to a leaf node in the tree, which canbe represented by a path stretching from the route.
Suppose that the K-ary tree is utilized, eachitem can be represented by a sequence of branch ID ( 0 , ..., K − 1 ), whose length is ⌈ log K N ⌉ .
Forexample, in Figure 2(b), the item i 4 is represented by “011” and “110” in the corresponding binarytrees, respectively.
Given the hierarchical numbering of the items, the recommendation turns out tobe a sequence-to-sequence problem: based on the encoded user representation, paths to the mostpreferable items are progressively decoded via beam search, from which the top-n recommendationis made.
The overall framework of RecForest is shown in Figure 2(a).
For ease of understanding, let’s first summarize the notations used in this paper.
Let x u denotethe user information of u , which can be either a behavior sequence or a user feature vector.
Lety i ∈ { 0 , K − 1 } H be the item i ’s routing sequence of length H and Y i ∈ { 0 , K − 1 } T × H be aconcatenation of item i ’s routing sequences of T trees.
Formally, RecForst models the conditionalprobability P
( Y i | x u ) with the seq2seq architecture.
2.3 Construction of Tree and Forest
As mentioned, the item set is organized by a tree structure, where semantically close items shouldfall into the same branch as much as possible.
We will construct multiple trees so as to alleviate thefalse assignment of items around partition boundaries.
Detailed workflows are illustrated as follows.
2.3.1 Tree Construction
The tree is expected to be balanced for the sake of efficient beam search on the tree.
Thus, we leveragehierarchical balanced clustering, where each cluster is evenly divided into K child clusters untileach child cluster merely contains one single item.
We propose to randomly sample K H − N items( H = ⌈ log K N ⌉ is the height of the tree) from the entire item set, such that the constructed tree willbe a complete K-ary tree.
We suggest the following two methods for the balanced partition of the item set.
Random :
For eachcluster, the included items are randomly partitioned into K equally-sized subsets.
Such a methodis extremely simple; yet, the semantic relationship between items is ignored.
Balanced Kmeans(short for Kmeans): Particularly, we first pre-train item embeddings based on an arbitrary off-theshelf recommendation model.
In our experiment, we use the Deep Interest Network (DIN) [20] forsequential recommendation and the Bayesian Personalized Ranking (BPR) [13] for non-sequentialrecommendation, given their effectiveness and popularity in corresponding scenarios.
With such arecommendation model, the items’ semantic closeness can be measured by their embedding similarity,as relevant items will be close to each other in the latent space.
Then, for each cluster, the includeditems are evenly partitioned via Kmeans w.r.t.
their embedding similarity.
Due to the space limitation,details about balanced Kmeans clustering are given in the appendix.
2.3.2 Forest Construction
Different trees are desirable of being diversified such that the boundary items can be better coveredfrom beam search.
To this end, we propose a simple but effective approach, which combines therandom and Kmeans tree construction.
Particularly, for each cluster with n items, n mod K itemsare first randomly chosen, from which one item is sampled with replacement into each child cluster;then, the remaining items are further partitioned into the K equally-sized groups via balanced Kmeans.
By doing so, different trees are naturally diversified due to the inherent randomness.
2.4 RecForest Enocoder
As shown in Figure 2(a), the encoder is utilized to encode user information and output user representation.
Any neural network based encoders are compatible with our framework.
We mainly considerthe sequential scenario and non-sequential scenario, where a sequence encoder and a feature encoderis used to encode user behavior sequences and feature vectors, respectively.
Feature Encoder
In the non-sequential scenarios, user information is denoted by a feature vector,i.e. x u .
We simply use a MLP for the feature encoder.
z u =
MLP ( x u ) .
When only user id isavailable, we can take its embedding as user representation, without any non-linear transformation.
Sequence Encoder
In the sequential scenarios, user information is an item sequence.
Concretely,x u =
[ i t 1 , i t 2 , . .
, i t m ] , where the item i t j ( 1 ≤ j ≤ m ) is the j -th interacted item of user u .
Itis straightforward to take item features into account.
In view of Transformer’s [18] state-of-theart performance in seq2seq tasks like machine translation and speech recognition, we utilize the
Transformer encoder for the sequence encoder.
In particular, z u = transformer-encoder ( x u ) , wherethe length of z u is the same as x u .
Due to space limitation, more details about the Transformerencoder are provided in the appendix.
2.5 RecForest Decoder
The decoder is to predict the next branch (routing decision) given the routing trajectory and theencoder’s outputs of user representation.
As shown in Figure 2(a), we apply the Transformer [18]decoder for this task due to its powerful capacity.
From down to up, following the branch embeddingand positional embedding, the Masked Multi-Head Attention [18] is utilized to model the complexdependence among branches at different layers.
Taking the outputs from the Masked Multi-Head
Attention, Multi-Head Attention [18] is followed to encode the complex interactions between userrepresentation and routing trajectory representations.
The FFN layer then enhances non-linearity ofcontext-aware routing trajectory representations.
A linear layer with the softmax activation is appliedfor predicting the next routing decision.
Note that this decoder can also handle the non-sequencescenarios since a user feature vector can be considered as a sequence only with one element.
2.6 Training
As depicted in Figure 2(b), the routing trajectory of item i 4 is [0,1,1] and [1,1,0] on the two trees,respectively.
The framework uses [start,0,1] to predict [0,1,1] and uses [start,1,1] to predict [1,1,0].‘start’ is a special symbol for initializing the sequence, which actually corresponds to the treeroot.
According to experimental results, we do not share the transformer decoder among trees, i.e.,
P ( Y i | x u ) =
(cid:81) k P θ k
( Y i [ k ] | x u ) , where θ k indicate the parameters of the k -th tree.
In this case, alltrees share the same training procedure, so we illustrate the training with one tree.
At each layer, thetraining task is to predict the next routing decision given routing trajectory, which corresponds to amulti-class classification problem.
Given a user u and its interacted item i , the loss for optimizingencoder and decoder is formulated as follows:
[Equation]
where y hi and y [0: h ] i indicate the branch (routing decision) at the h-th layer and routing trajectory untilthe h-th layer respectively, and ‘cat’ means concatenation.
2.7 Inference
Since each leaf in the tree corresponds to an item, RecForest transforms top-n recommendation into thesequence prediction problem.
In particular, we first apply the encoder for deriving user representationand leverage the decoder to generate the top-n routing sequences, i.e., paths from the root node toleaf nodes on each tree, based on beam search.
Note that each item can correspond to multiple leafnodes (i.e. routing sequences) in each tree since some items are randomly sampled for building acomplete K-ary tree.
Therefore, the generated items should be duplicated.
Let C denote the union ofgenerated candidate items from the forest.
For c ∈ C , we first compute the logarithmic probability 189 (cid:80) H −h =0 log Prob ( y hi | cat ( start , y [0: h ] i ) , z u ) in each tree and then sum logarithmic probabilities over alltrees as the score of the item c .
We then select the top-n items from C for recommendation.
2.8 Complexity Analysis
Notation Denote by D item embedding size, B the beam search size, K the number of branches,and I the training set size.
Denote by T the number of trees in RecForest and the times of updatingthe tree in JTM and TDM.
In SCANN, denote by K vq the number of centroids in VQ, K pq thenumber of centroids in PQ, M the number of subspaces and W the number of probed VQ cells.
Time Complexity In inference, the beam search can be finished within O ( TKB log K N ) sincethe probabilities of KB branches at each level of all trees need to be calculated.
Hierarchical Kmeansclustering can be done within O ( NK log K N ) to build a tree index so that it takes O ( TNK log K N )to build the forest index.
Space Complexity
The memory consumption of the recommendation model is inevitable in anyrecommender systems, so we mainly focus on the size of the index structure.
In RecForest, we onlyneed to keep the tree structure and T matrices with size K × D , denoting branch embeddings of Ttrees.
In TDM and JTM, beside the tree structure, they also store all tree node embeddings, where thenumber of tree nodes is at the same magnitude as the number of the items.
[Table]
We summarize complexity analysis of typical algorithms in Table 1, which shows RecForest enjoys asmall index memory cost and low inference time.
This is because a smaller beamsize and fewer beamsearch are only required, and tree node representations are not stored but computed on the fly.
Experiments
We conduct the experiments to answer the following research questions: RQ-1 : Does RecForestoutperform the SOTA efficient recommenders in the tradeoff between efficiency and accuracy?
RQ2 : Does the forest-based index improve near-boundary item retrieval? RQ-3 : How much effectpositional embedding can take?
The experiments are carried out in both sequential scenarios andnon-sequential scenarios, where the sequence encoder and feature encoder is utilized, respectively.
Due to space limitations, we only report the results in the sequential scenario.
Other results areprovided in the appendix.
These experiments are done on a Linux server with Tesla V100 GPUs.
3.1 Dataset
[Table]
We evaluate the RecForest with six real-worldrecommendation datasets, which can be downloaded from the url 1 .
The datasets are MovieLens 10M (abbreviated as Movie ), Amazon
Books (abbreviated as Amazon ), Tmall Click(abbreviated as Tmall ), Gowalla Check-in
Dataset (abbreviated as Gowalla ), Microsoft
News Dataset (abbreviated as MIND ).
Sincesome datasets only include rating-based explicitfeedback, they should be converted into implicitfeedback for RecForest’s inputs.
These datasetsare pre-processed by filtering the users who interact with no more than 15 items.
The overallinformation of datasets is summarized in Table 2.
3.2 Baselines
In the part, we report the results of the sequential recommendation scenario, so we compare theproposed RecForest with two-stage indexes, IPNSW [10] and SCANN [3], with learnable indexes,
TDM [23] and JTM [22] as well as with YoutubeDNN [1] and DIN [20] based on the brute-forceretrieval.
Note that brute-force-based YoutubeDNN and DIN are time-prohibitive in online services,but they can be considered as two strong baselines.
IPNSW and SCANN are built over itemembedding of a recommender.
The recommender shares the same encoder as RecForest, but replacesthe decoder with flat item embedding. It is then trained with the same loss as DIN.
Both TDM and JTM make a routing decision simply based on current node without the consideration of routingtrajectories.
More details about baselines can be referred to in the appendix.
3.3 Experimental Settings
In each dataset, we randomly choose 10% users as validation users, 10% users as test users, and allthe left users as training users.
Following TDM and JTM, we use a slide window to split user-iteminteraction histories into slices of length 70 at most.
For training users’ data, the first 69 interactionsare used for input context and the 70-th item is regarded as the ground truth of prediction.
For data ofboth validation users and test users, we regard the first half as context and others as ground truth.
The latent dimensionality is set to 96 in all methods.
Here, for a fair comparison of running time, weimplement SCANN and IPNSW with PyTorch.
To construct the quantization-based index, we follow
SCANN’s default settings.
To construct the graph index of IPNSW, the maximum degree of eachnode is set to 16.
The beam size for any beam search is set to 100 unless specified.
To learn models,the learning rate is all set to 1e-3 with exponential decay.
The items’ representations for constructingtrees are obtained from item embedding of the pre-trained DIN on each dataset.
3.4 Comparison with Baselines
Settings We compare RecForest with the baselines on all datasets.
RecForest uses 2 trees on the
Movie and MIND, and 4 trees on other datasets.
Other settings can be referred to in Section 3.3.
[Table]
Results All the results are shown in Table [Table] where the index memory cost of RecForest indicatesthe decoder’s memory consumption.
The following observations answers the RQ-1 .
• RecForest significantly outperforms all efficient recommenders with indexes on all datasets w.r.t NDCG@20 and NDCG@40 .
The improvements over the best baselines w.r.t NDCG@20are 4.69%, 3.72%, 45.78%, 21.40%, 3.23%, 7.96% on the Movie, Amazon, Gowalla, Tmall,MIND, and Yelp dataset, respectively.
Its superiority over two-stage baselines (i.e., IPNSW andSCANN) demonstrates the effectiveness of joint representation learning, although the graph-basedindex shows a strong performance.
The higher accuracy than TDM and JTM indicates the benefitfrom the use of routing trajectory, multiple trees and the powerful transformer decoder.• RecForest performs better than brute-force-based YoutubeDNN on all datasets and eventhan brute-force-based DIN on the Movie, Gowalla, and MIND datasets.
Note that wedo not apply DIN for post-reranking, otherwise the retrieval accuracy can be further improved.
This again confirms the superpower of the transformer decoder.
Note that after fine-tuning thehyper-parameters, YoutubeDNN performs well compared to TDM and JTM, which may not beconsistent with the results in TDM and JTM.• RecForest also enjoys the smallest index memory cost compared to other methods , which isnow supported empirically and theoretically.
This is because RecForest only embeds the routingdecision and compute tree node representation on the fly, compared to directly embedding alltree nodes in TDM and JTM.
RecForest even costs fewer memories for index than the mostlightweight PQ-based index – SCANN, which only stores the codes of data points and codebooks.
In spite of good retrieval accuracy, the graph-based index IPNSW is memory-heavy due to thestorage of graphs and data points.
Note that both DIN and YoutubeDNN are based on exhaustivesearch so they don’t consume memory for indexes. IPNSW SCANN JTM
TDM
RecForest 1 2 3 4 5 6 7 8 Time(s) 0.
{+N D C G @ 
+}
Amazon 1 2 3 4 5 6 7 8 9 Time(s) 0.50 0.
MIND 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Time(s) 0.
Gowalla 1 2 3 4 5 6 7 Time(s) 0.
Yelp
• RecForest is almost as fast as and sometimes even faster than SCANN for the top-n itemretrieval , which is considered as the SOTA index for MIPS in retrieval efficiency and accuracy.
This benefits from transforming the top-n recommendation into sequence prediction, which canbe efficiently accomplished by beam search on GPU.
Note that the running time cost of IPNSWseems inconsistent with Table 1, this is because only the approximate proximity graphs instead ofexact Delaunay graphs can be efficiently constructed for query in practice.
Theoretical resultsshow that the RecForest is almost as efficient as TDM and JTM, while the experimental resultsindicate RecForest is slightly faster.
On the one hand, this is because we use the comparativelylarge branches (much larger than 2 used in TDM and JTM), remarkably decreasing the tree depthand thus reducing the times of beam search.
On the other hand, because RecForest uses the simplefunction for routing decision and enables high parallelism due to the use of transformer decoder.
3.5 Extensive Study between Efficiency and Accuracy
Settings
The efficiency-accuracy curve is a commonly-used standard for evaluating the ANNsindex, so we also provide this result for better illustrating the superiority of RecForest.
The study ismainly investigated on the Amazon, MIND, Gowalla, and Yelp dataset, since the other two datasetsshow a similar trend.
To vary the retrieval time of RecForest, we adjust the beam size of beam searchfrom 10 to 100 with step 10, where the forest consists of at most 5 trees.
For TDM, JTM, and IPNSW,we adjust the beam size from 10 to 200 with step 10.
For SCANN, we adjust the number of probecells from 50 to 2,000 with step 100.
Results
The curves of different algorithms between NDCG@20 and query time are shown in Figure3.
The following observations confirms the answer of RQ-1 .
First, RecForest strikes the bestbalance between query time and retrieval accuracy on all the four datasets , since the curvestands on top of the others.
This confirms the superiority of RecForest to these competing baselines.
The advantage is the most significant on the Gowalla dataset, evidenced by the biggest gap between
RecForest and IPNSW.
Second, With the increase of beam size, the accuracy of RecForest canimprove more significantly than baselines.
This benefits from the powerful transformer decoderand the novel training paradigm as well as the forest-based index, such that the representation modeland tree indexes are jointly learned better.
3.6 Ablation Study
3.6.1 Effect of Forest Construction
Settings We investigate three aforementioned ways (in Section 2.3.2) of constructing a forest withat most 10 trees, where the branch number of each tree is set to 4.
Results
The results on the Amazon, Gowalla, MIND, and Yelp datasets are reported in Figure 4.
We can observe that Kmeans with randomness (i.e. Random+Kmeans) always performs bestand improves with the increasing number of trees while the Random performs worst and doesnot improve as much as Random+Kmeans when the tree number increases.
The reason why the
Random does not work well is that the semantic requirement is not satisfied.
Since Kmeans utilizesthe semantic information, it performs better than the Random, but is not affected a lot by growingthe number of trees.
This can be explained by the lack of diversity between trees.
This answers the
RQ-2 : since the use of forest-based index can lead to significant improvement of retrieval accuracy,particularly when tree construction takes semantics into account, the forest-based index improvesnear-boundary item retrieval .
Random
Kmeans
Random+Kmeans 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
N D C G @
Amazon 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
MIND 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
Gowalla 1 2 3 4 5 6 7 8 9 10 The number of trees 0.
Yelp
3.6.2 Effect of Branch Number K
Settings We conduct evaluations on a single Kmeans-constructed tree on the Amazon, MIND,
Gowalla and Yelp dataset, by varying the branch number in {2,4,8,10,16,18}.
[Table]
Results The results are shown in Table 4.
We can observe that the retrieval accuracyof RecForest improves with the growingnumber of branches, particularly from2 branches to 4.
The improvement on the Gowalla and Yelp datasets is the most significant.
This observation can be explainedas follows:
First, the vocabulary size grows,so the decoder can be supervised by morechallenging discriminative signals and thedecoders’ parameters grow.
Secondly, therouting sequences would be shorted, reducing the error accumulation of sequence prediction to some extent.
3.6.3 Effect of Positional Embedding
Settings Since the routing sequence is used for representing an item, the order of routing decisionsis important.
To this end, we compare four positional embeddings: (1) None indicates that there isno positional embedding; (2) Absolute positional embedding ( Abs for short) encodes the absolutepositions which range from 1 to maximum length of sequence by learnable parameters; (3) Relative
Key [14] ( RelK for short) focuses on attention and relative distance between decisions; (4) Relative
Key Query [6] ( RelKQ for short) is a refined Relative Key.
Please refer to [6] for more details.
[Table]
Results
The results based on from single
Kmeans-constructed tree on the Amazon,
MIND, Gowalla, and Yelp datasets are summarized in Table 5.
We can observe that positional embedding takes a remarkableeffect on improving the retrieval capacity while different positional embeddingmethods do not have a significant difference in retrieval accuracy , which answersthe RQ-3 .
The effects of positional embedding are more significant on the Gowalla and Yelp datasets.
Conclusion and Future Work
In this paper, we propose the Recommender Forest for efficient recommendation, which can be simplytrained within the sequence-to-sequence framework.
RecForest enjoys a small index memory cost,low inference time, and highly-accurate recommendation even without updating the tree structure.
The extensive study on six real-world recommendation datasets shows that RecForest becomes a stateof-the-art efficient recommender.
In the future, we explore multi-task learning, non-autoregressiveprediction, and index structure learning as well as a more general framework.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please see the future work insection 4.(c)
Did you discuss any potential negative societal impacts of your work?
[No] For theproposed RecForest model, we have not seen negative societal impacts(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide the
URL in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see section 3.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendices
for the Submission: Recommender Forest for Efficient Retrival
Our appendices include the following content:
• Sec.
A: the details about forest construction.
• Sec.
B: the illustration of Transformer encoder.
• Sec.
C: the pseudocode of training and inference.
• Sec.
D: the metrics used in performance evaluation.
• Sec.
E: the experiment details in sequential recommendation scenario. • Sec.
F: the experiments conducted in non-sequential scenarios.
A Forest Construction
A.1 Balanced Kmeans
For the sake of retrieval efficiency, the constructed tree is expected to be a complete K-ary tree.
However, the item number N does not necessarily meet this requirement.
We grow the dataset byinserting K H − N randomly sampled items from itself, such that it exactly contains K H items, where
H = ⌈ log K N ⌉ is the tree height.
We use the Balanced Kmeans in Algorithm 1 for hierarchicalitemset partition.
In particular, supposing a cluster contains K × m items, the algorithm splits theminto K clusters such that each cluster exactly contains m items.
At each iteration, we first assigneach centroid the m closest unassigned items in order and then update the centroids according toitem assignment.
The clustering algorithm terminates until the assignments do not change any more.
Algorithm 1: Balanced-Kmeans
Data: Item set D , cluster number K
Result: { D 1 , D 2 , . . .
, D K } 1 m ← | D | K ; 2 Randomly initialize K centroids C
= { c 1 , c 2 , . .
. , c K } ; 3 repeat
U ← D ; for k = 1 → K do
U s ← Sort items in U ascendingly according to distance from c k ;
D k ← U s [0 : m − 1] ; c k ← 1 m (cid:80) x ∈ D k x ;
U ← U \ D k ; 10 until Assignments do not change ;
A.2 Tree Construction
As described in Section 2.3.2, each tree is constructed by combining Random partition and Balanced
Kmeans to guarantee both diversity and semantic closeness of similar items.
Regarding the Randomstrategy, for each cluster, the included items are randomly partitioned into K equally-sized subsets.
When it is combined with balanced Kmeans, for each cluster with n items, n mod K items arefirst randomly picked, from which one item is sampled with replacement into each child cluster,and the remaining items are partitioned based on the Balanced Kmeans.
This method is shorten for
Random+Kmeans .
By applying the Random+Kmeans strategy recursively until each cluster onlycontains one item, the tree index can be constructed successfully.
Repeating the tree construction cangenerate diverse trees to form the forest.
We summarize the procedure in Algorithm 2.
B Transformer encoder
The Transformer encoder, shown in Figure 5, is to capture user’s interests from the sequential interactions x u =
[ i t 1 , i t 2 , . .
, i t m ] .
From down to up, following the interaction embedding and positional
Algorithm
Data: Dataset D , branch number K , random ratio δ
Result: Constructed tree 1 Q ← A queue with an element D ; 2 while Q is not empty
do
D top ← pop( Q ); n ← | D top | ;
D 0 ← Randomly draw n mod K items from D top without replacement ;
D top ← D top \ D 0 ; { D 1 , D 2 , . . .
, D K } ← Balanced-Kmeans( D top , K ); for k = 1 → K do i ← Sample one item with replacement from D 0 ; push( Q , { i } ∪ D k ); embedding, Multi-Head Attention [18] is utilized to model the interdependence of user’s each behavior.
Taking the output of Multi-Head Attention, the FFN layer is to enhance the non-linearity ofuser’s behavior representations.
Finally, the encoder output, denoted as z u =
[ z 1 u ; z 2 u ; . . .
; z mu ] , isutilized to characterize user representation.
[Figure]
C Pseudocode of training and inference
In our RecForest, all the trees share the encoder but each tree has a unique decoder.
The trainingframework is summarized in Algorithm 3.
Note that we only give one training instance for clearillustration and it can be done in batch-wise manner for practical implementation.
In addition, theinference framework is summarized in Algorithm 4.
Once the generated candidates from each treeare collected, we compute the generated probabilities of candidates on each tree for ranking purpose.
Data: Forest = { tree 1 , tree 2 , . .
. , tree T } , training instance ( x u , i ), Encoder and Decoder t ( 1 ≤ t ≤ T ) w.r.t.
each tree 1 z u ← Encoder( x u ); 2 foreach t ∈ { 1 , 2 , . .
, T } do target ← get the path of item
i on the tree t ; decoder-input ← [‘start’,target[0:H-1]]; // H is tree height prediction ← Decoder t ( z u ,decoder-input);
L ( u, i ) ← loss(prediction, target[1:H]) according to Eq (1); back-propagation from L ( u, i ) ; update Encoder and Decoder t by gradient descent;
Data: Forest = { tree 1 , tree 2 , . .
. , tree T } , Encoder and Decoder t ( 1 ≤ t ≤ T ) w.r.t.
each tree, user feature x u , beam size B , n (i.e. number of returning items)
Result:
The top- n items for user u 1 R ← {}; 2 z u ← Encoder( x u ); 3 for t ∈ { 1 , 2 , . . .
, T } do paths ← Beam-Search(Decoder t , z u , B ) ; foreach p ∈ paths do item i ← get-item( p, tree t );
R ← R ∪ { i } ; 8 foreach item i ∈ R do score [ i ] = 0 ; for t ∈ { 1 , 2 , . .
, T } do path ← get-path( i, tree t ); p t ( i ) ← compute the path probability based on Decoder t ; score [ i ] = score
[ i ] + log p t ( i ) ; 14 select the top- n items from R based on score ;
D Metrics
The ranking metric, Normalized Discounted Cumulative Gain( NDCG ), is used to measure theperformance of recommendation engine.
Discounted Cumulative Gain( DCG ) can be calculated by
[Equation]
where r i means the relevance score and k is the number of returned items.
As we focus on implicitfeedback, r i = 1 if the test user really interacts with the returned i -th item otherwise r i = 0 .
NDCGcan be calculated by
[Equation]
where iDCG means ideal DCG , i.e. ranking all the returned items according to relevance scores ina descending order and calculating the DCG on the ordered items.
Additionally, we also concern thememory consumption of retrieval index and inference efficiency.
Concretely, we record the modelsize of retrieval system and the time consumption in inference.
E Experiment Details in sequential recommendation scenarioE.1 Baselines in sequential recommendation scenario
• TDM [23] and its variant JTM [22], proposed by Alibaba Group, are the tree-based recommender
frameworks which train the user-item preference model and the tree structure simultaneously.
The
leaf nodes represent items and the internal nodes represent the clusters which the correspondingleaf nodes belong to.
Beam search is conducted on the tree to retrieve the items from coarse-grainclusters to fine-grain clusters.
The main difference between TDM and JTM is the updating ofbijective mapping relation between items and leaf nodes.•
Deep Interest Network (i e. DIN ) [20] utilizes the deep model to characterize the user’s preferencesover items from user’s sequential interactions.
Concretely, DIN casts the training task as a binaryclass classification that reflects whether the user interacted with certain item.
As DIN can’t beused for efficient recommendation, we report the brute-force-based performance.• YoutubeDNN [1], proposed by Google, is a deep neural network model which learns the userrepresentation and item representation from sequential user-item interaction history.
Then the innerproduct between user representation and item representation is used to capture user’s preferenceover the corresponding item, which converts the recommendation task to maximum inner productsearch(MIPS).
In our experiments, we mainly show the performance of YoutubeDNN by burteforce search for MIPS.• To solve MIPS efficiently, Quantizaiton-based method SCANN [3] and graph-based methodIPNSW [10] meet the start-of-the-art performance in each technique route respectively.
For faircomparison with our model, we model user’s interactions by Transformer encoder and replace thedecoder by item embedding.
The adapted model is trained in the same way as DIN to obtain theuser representation and item representation so that the learned representation can be used to buildthe SCANN index and IPNSW index.
The open-source code of TDM 2 , JTM 3 and DIN 4 are based on Alibaba Group’s X-DeepLeraning framework, we re-implement them by Pytorch.
And the open-source code of YoutubeDNN 5 is alsoavailable.
E.2 Detailed settings in sequential recommendation scenario
In each dataset, we randomly choose 10% users as validation users, 10% users as test users and allthe left users as training users.
Following the settings in TDM [23] and JTM [22], we use a slidewindow to cut user-item interaction histories into slices of length at most 70.
Each window containsat least 15 interactions and zero padding is applied if there is less than 70 interactions in the window.
At the training stage, the first 69 user-item interactions are regarded as context and the 70-th item isregarded as ground truth in each window.
For validation users and test users, we regard the secondhalf interactions as ground truth and the first half interactions as context.
The hidden size is set to 96for all methods.
The Deep Interest Network (DIN) [20], which contains the embedding layer followed by a MLPwith size [128,64,2] and outputs the like/dislike probability, is used as the the preference model for TDM and JTM.
YoutubeDNN [1] contains embedding layer followed by a MLP with size [128,64,96]and the number of negative sampling is set to 1,000.
For SCANN, we follow the default settingsof the open sources 6 but implement them with Python and Pytorch for fair efficiency comparison.
Concretely, the number of leaf is 2,000; the number of sub-space is 24 and the dimensionality of eachsub-space is 4; the number of codewords in each sub-space is set to 16; the value of threshold is set to0.2.
To construct the graph index of IPNSW, the maximum degree of each node is set to 16, and thebeam size is set to 100.
For our RecForest, YoutubeDNN, TDM, JTM and DIN, Adam [7] is used as the optimizer andlearning rate is set to 1.0e-3 with exponential decay.
In inference, the beamsize of RecForest, TDM,
JTM and IPNSW is set to 100 unless specification. For RecForest, the branch number is set to 18 andthe tree number is set to be 2 on Movie and MIND, and 4 on the left datasets unless specification. The items’ representations which are utilized to construct the forest for RecForest are obtained by pre-train the DIN model on each dataset.
F Experiments in non-sequential scenarios
In non-sequential scenarios, we only apply user id to denote a user and the feature encoder tomodel user representation.
Specifically, the learnable embedding vector of the user id denotes userrepresentation.
F.1 Baselines
The classical deep model, Neural Collaborative Filtering ( NCF ) [4], characterizes user’s preferenceover item given the user id and item id. Due to the compatibility with any advanced user-item (oruser-node) preference model, we utilize NCF for preference modeling in TDM [23] and JTM [22]frameworks.
For YoutubeDNN , we replace the user representation learning part used in sequentialrecommendation scenarios with the embedding vector of user id.
By this way, YoutubeDNN canbe compatible with non-sequential recommendation task.
SCANN and IPNSW are built on thelearned representation of YoutubeDNN.
Note that brute-force-based YoutubeDNN and NCF aretime-prohibitive in online services, but they can be considered as two strong baselines.
F.2 Experimental settings
In each dataset, we randomly choose 10% users as validation users, 10% users as test users and allthe left users as training users.
For validation users and test users, we randomly select a quarter of theinteracted items as ground truth, and put the rest into the training set.
The fusion NCF [4], whichcontains the GMF and MLP where the neuron numbers are set to [512,256,128, 96] in each layer, isincluded.
Other necessary settings are as described in Section E.2.
F.3 Comparison with Baselines
Settings We compare RecForest with the baselines on all datasets.
RecForest uses 2 trees on the
Movie and MIND, and 4 trees on other datasets.
Other settings can be referred to in Section F.2.
Results All the results are shown in Table 6, where the index memory cost of RecForest indicatesthe decoder’s memory consumption.
From the table, we have the following findings.
• Recforest nearly outperforms all efficient recommenders with indexes on all dataset w.r.t.
NDCG@20 and NDCG@40 except that SCANN outperforms RecForest slightly on Moview.r.t NDCG@40.
Its superiority over two-stage baselines is not so significant indicates thatinner product may be capable enough for representation learning to some extent in this simplerecommendation scenario.
The higher accuracy than TDM and JTM also verify the benefit fromthe use of routing trajectory, multiple trees and the powerful transformer decoder.• RecForest performs better than brute-force-based NCF and YoutubeDNN on nearly alldatasets except the results on Tmall where YoutubeDNN outperforms RecForest slightly.
Thisphenomenon confirms the the superpower of transformer decoder.• Being similar to the results of sequential recommendation scenarios, RecForest has the advantage in index memory cost , which can be highlighted by comparisons with baselines.
Theseobservations can be explained by referring to the same experiments in sequential scenarios.
Notethat NCF and YoutubeDNN are based on exhaustive search so they don’t consume memory forindexes.
F.4 Extensive Study between Efficiency and Accuracy
Settings Similar to settings in sequential recommendation scenarios, we also draw the efficiencyaccuracy curve.
The study is mainly investigated on the Amazon, MIND, Gowalla, and Yelp dataset,since the other two datasets show a similar trend.
As described in the sequential scenario, we adjustthe beams size to vary the retrieval time cost for TDM, JTM, IPNSW and RecForest.
For SCANN,we adjust the number of probe cells to control the time cost.
Results
The curves of different algorithms between NDCG@20 and query time are shown in Figure 6.
From the figure, we have the following findings.
[Table] [Table]
[Figure]
• RecForest strikes the best balance between query time and retrieval accuracy on all the four
datasets , since the curve stands on top of the others.
This confirms the superiority of RecForest
to these competing baselines.
The advantage is the most significant on the MIND, evidenced by
the biggest gap between RecForest and IPNSW.
• With the increase of beam size, the accuracy of RecForest can improve more significantly
than baselines.
This benefits from the powerful transformer decoder and the novel training
paradigm as well as the forest-based index, such that the representation model and tree indexes
are jointly learned better.
F.5 Ablation Study
F.5.1 Effect of Forest Construction
Settings We investigate three aforementioned ways of constructing a forest with at most 10 trees,where the branch number of each tree is set to 4.
The details about forest construction can be referredto in Section 2.3.2.
Results
The results on the Amazon, MIND, Gowalla and Yelp datasets are reported in Figure 7.
Theresults on Amazon, MIND and Yelp show the same trends as ones in sequential scenarios.
Concretely,
Kmeans with randomness (i.e. Random+Kmeans) always performs best and improves with theincreasing number of trees while the Random nearly performs worst and does not improve asmuch as Random+Kmeans when the tree number increases.
These results verify the findingsagain, i.e. the semantic information and the diversity among trees are important to improve therecommendation accuracy.0.04 0.02 0.00 0.02 0.
[Figure]
F.5.2 Effect of Branch Number K
Settings We conduct evaluations on a single Kmeans-constructed tree on the Amazon, MIND,
Gowalla and Yelp dataset, where the branch number varies in {2,4,8,10,16,18}.
[Table]
MIND
[Table]
Yelp 0.
Results The results are shown in Table 7.
We can observe that the retrieval accuracyof RecForest on Amazon improves withthe growing number of branches constantly but does not show a clear trendon other three datasets.
These findings indicate that the sensitivity of branch numberdepends on the dataset in non-sequentialrecommendation scenarios.
Generally, theincrease of branch numbers and the decrease of path length don’t affect the performance as much as that in the sequentialscenario.
F.5.3 Effect of Positional Embedding
Settings We also investigate the effect of four positional embedding methods like what we do inthe sequential recommendation scenario.
Concretely, (1) None ;(2) Abs ;(3) RelK ;(4) RelKQ and moredetails are referred to in Section 3.6.3.
[Table]
Results
The results based on a single
Kmeans-constructed tree on the Amazon,
MIND, Gowalla, and Yelp datasets are summarized in Table 8.
It is similarly observedto Section 3.6.3 that positional embeddingtakes a remarkable effect on improvingthe retrieval capacity while different positional embedding methods do not havea significant difference in retrieval accuracy .
The effect of Abs and RelK is moresignificant on Gowalla.