On the Generalizability and Predictability of Recommender Systems
Abstract
While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort.
Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines.
A natural follow-up question is, “how do we choose the right algorithm for a new dataset and performance metric?”
In this work, we start by giving the first large-scale study of recommender system approaches by comparing 24 algorithms andsets of hyperparameters across 85 datasets and 315 metrics.
We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric, however, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets.
Motivated by these findings, we create RecZilla, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets.
By using far more meta-training data than prior work,
RecZilla is able to substantially reduce the level of human involvement when faced with a new recommender system application.
We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, so that practitioners can train a RecZilla model for their desired performance metric: https://github.com/naszilla/reczilla .
Introduction
Due to the large computational resources for training machine learning models, researchers have found many ways to repurpose existing computation.
For example, it is common to start with a pretrained ImageNet classification model for computer vision tasks [30, 50, 75], or a pretrained BERT model for natural language tasks [31, 65].
These approaches work well because the core problems are largely homogeneous; for example, any computer vision model at its core must be able to distinguish edges, colors and shapes.
Even a task-specific architecture can be found automatically through neural architecture search [36], since the building blocks such as convolutional layers stay the same.
On the other hand, recommender system (rec-sys) research has followed a different trajectory: despite their widespread usage across many e-commerce, social media, and entertainment companies such as
Amazon, YouTube, and Netflix [21, 41, 77], there is far less work in reusing models or automating the process of selecting models.
Many rec-sys techniques are designed and optimized with just a single dataset in mind [21, 41, 48, 58, 81].
Intuitively, this might be because each rec-sys application is highly unique based on the dataset and the target metric.
For example, a typical user session looks very different among e.g. YouTube, Home Depot, and AirBnB [21, 48, 58].
However, this intuition
RecZilla for Prediction (Online)
RecZilla Training Pipeline (Offline)
Algorithm Selection
Select n parameterized rec-sys algorithms with high coverage for metric on datasets in
Dataset Meta-feature Selection
Select m dataset meta-features which are correlated with metric for the selected algs.
Meta-Model training
Learn function that maps meta-features to metric for all selected algs.
Calculate Meta-features
Calculate m
[Figure]
for dataset D
User input
Performance metric: A function of the metrics in the meta-dataset
New rec-sys dataset: D
[Figure]
Output
[Figure]
has not been established system algorithms do not always improve over well-tuned baselines matrix factorization [
In this work, we show that the best algorithm and hyperparameters are highly dependent on the dataset and user-defined performance metric.
Specifically, we run the first large-scale study of rec-sys approaches by comparing 24 algorithms across 85 datasets and 315 metrics.
For each dataset and algorithm pair, we test up to 100 hyperparameters (given a 10 hour time limit per pair).
The codebase that we release, which includes a unified API for a large, diverse set of algorithms, datasets, and metrics, may be of independent interest.
We show that the algorithms do not generalize – the set of algorithms which perform well changes substantially across dataset and across performance metrics.
Furthermore, the best hyperparameters of a rec-sys algorithm on one dataset often perform significantly worse than the best hyperparameters on a different dataset.
Although we show that there are no universal algorithms that work well on most datasets, we do show that various meta-features of the dataset can be used to predict the performance of rec-sys algorithms.
In fact, the same metafeatures are also predictive of the runtime of rec-sys algorithms as well as the “dataset hardness” – how challenging it is to find a high-performing model on a particular dataset.
Motivated by these findings, we introduce RecZilla, a meta-learning-based algorithm selection approach (see Figure 1) inspired by SATzilla [83].
At the core of RecZilla is a model that, given a user-defined performance metric, predicts the best rec-sys algorithm and hyperparameters for a new dataset based on meta dataset features such as number of users and items, and spectral properties of the interaction matrix.
We show that RecZilla quickly finds high-performing algorithms on datasets it has never seen before.
While there has been prior work on meta-learning for recommender systems [26, 27], no prior work is metric-independent, searches for hyperparameters as well as algorithms, or considers more than nine dataset families.
By running an ablation study on the number of metatraining datasets, we show that more dataset families are crucial to the success of RecZilla.
We release ready-to-use, pretrained RecZilla models for common test metrics, and we release the raw results from our large-scale study, along with code so that practitioners can easily train a new RecZilla model for their specific performance metric of interest.
Our contributions.
We summarize our main contributions below.
• We run a large-scale study of recommender systems, showing that the best algorithm and hyperparameters are highly dependent on the dataset and user-defined performance metric.
We also show that dataset meta-features are predictive of the performance of algorithms.
• We create RecZilla, a meta-learning-based algorithm selection approach which, given a performance metric, efficiently predicts the best algorithm and set of hyperparameters on new datasets.
• We release a public repository containing 85 datasets and 24 rec-sys algorithms, accessed through a unified API.
Furthermore, we release both pretrained RecZilla models, and raw data so that users can train a new RecZilla model on their desired metric.
Related Work Recommender systems are a widely studied area of research [10].
Common approaches include k -nearest neighbors [1], matrix factorization [57, 63], and deep learning approaches [21, 41, 77].
For a survey on recommender systems, see [4, 10].
A recent meta-study showed that of the 12 published neural rec-sys approaches published at top conferences between 2015 and 2018, 11 performed worse than well-tuned baselines (e.g. nearest neighbor search or linear models) [28].
Another recent paper found that the relative performance of rec-sys algorithms can change significantly based on the choice of datasets used [14].
Algorithm selection for recommender systems was first studied in 2011 [52] by using a graph representation of item ratings.
Follow-up work used dataset meta-features to select the best nearest neighbor and matrix factorization algorithms [3, 35, 43].
Subsequent work focused on improving the model and framework [27] including studying 74 meta-features systematically [23].
More recent approaches from 2018 run meta-learning for recommender systems by casting the meta-problem itself as a collaborative filtering problem.
Performance is then estimated with subsampling landmarkers [24, 25, 26].
No prior work in algorithm selection for rec-sys includes open-source Python code.
There is also work on automated machine learning (AutoML) for recommender systems, without meta-learning [6, 46, 47, 82].
Finally, we note that meta-learning across rec-sys datasets is not to be confused with the body of work on meta-learning user preferences within a single rec-sys dataset [17, 18, 19].
To the best of our knowledge, no meta-learning or AutoML rec-sys paper has run experiments on more than nine dataset families or four test metrics, and no prior work predicts hyperparameters in addition to algorithms.
Analysis of Recommender Systems
In this section, we present a large-scale empirical study of rec-sys algorithms across a large, diverse set of datasets and metrics.
We assess the following two research questions.
Generalizability.
If a rec-sys algorithm or set of hyperparameters performs well on one dataset and metric, will it perform well on other datasets or on other metrics? 2.
Predictability.
Given a metric, can various dataset meta-features be used to predict the performance of rec-sys algorithms?
Algorithms, datasets, and metrics implemented.
We present full results for 20 rec-sys algorithms, including methods from recent literature and common baselines.
Methods include five similarity and clustering-based methods: User-KNN [74], Item-KNN [76], P3Alpha [20], RP3Beta [72], and
Co-Clustering [39]; six Matrix-Factorization (MF) methods: MF-FunkSVD, MF-AsySVD [56],
MF-BPR [73], IALS [51], Pure-SVD, Non-negative matrix factorization (NMF)
[22]; five methods based on linear models: Global-Effects, SLIM-BPR [7], SLIM-Elastic-Net [62], EASE-R [78], and
SlopeOne [61]; two simple baselines: Random and Top-Pop; and two neural network based methods:
User-NeuRec [84] and Mult-VAE [64].
We also include partial results (on 8-10 datasets each) for four more neural network based methods: DELF-EF [13], DELF-MLP [13], Item-NeuRec [84], and
Spectral-CF [85] (with results included in Tables 9, 10, and 11).
These algorithms were chosen due to their high performance, popularity, and speed.
For many algorithms, we used the implementations from the codebase of Dacrema et al.
For full details of the algorithms, see Appendix A.
We run the algorithms on 85 datasets from 19 dataset “families”:
Amazon [71], Anime [16],
BookCrossing [87], CiaoDVD [45, 59], Dating (Libimseti.cz) [59, 60], Epinions [67, 68], FilmTrust [44], Frappe [8], Gowalla [15], Jester2 [40], LastFM [11], MarketBias-Electronics and MarketBiasModCloth [80], MovieTweetings [33], Movielens [49], NetflixPrize [9], Recipes [66], Wikilens [37], and Yahoo [34].
Here, a “dataset family” refers to an original dataset, while “dataset” refers to a single train-test split drawn from the original dataset, which may be a small subset of the original.
We implemented the majority of rec-sys datasets commonly used for research; to the best of our knowledge, this is the largest number of rec-sys datasets accessible in a single open-source repository.
[Table]
We use 23 different “base” metrics: ARHR, Average Popularity, Diversity Similarity, F1 Score, Gini
Index, Herfindahl Index, Hit Rate, Item Coverage, Item-hit Coverage, Precision (PREC), PrecisionRecall Min Denominator, Recall, MAP, MAP Min Denominator, Mean Inter-List Diversity, MRR,
NDCG, Novelty, Shannon Entropy,
Coverage,
All of these metrics are computed at
{ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 15 , 20 , 30 , 40 , 50 } , for a total of 315 different metrics.
See Appendix A.6 for more details.
These metrics include the most popular from the literature, and we use the Dacrema et al.
[28] implementations.
Experimental design.
Each dataset’s train, validation, and test split is based on leave-last-
k -out (and our repository also includes splits based on global timestamp).
We use a random hyperparameter search for all methods, with the exception of neural network based methods.
Since neural networks require far more resources to train (longer training time, and requiring GPUs), we use only the default hyperparameters for neural algorithms.
For each non-neural algorithm, we expose several hyperparameters and give ranges based on common values.
For each dataset, we run each algorithm on a random sample of up to 100 hyperparameter sets.
Each algorithm is allocated a 10 hour limit for each dataset split; we train and test the algorithm with at most 100 hyperparameter sets on an n1-highmem-2 Google Cloud instance, until the time limit is reached.
Each neural network method is trained on each dataset using the default hyperparameters used in its respective paper, with a time limit of 15 hours on an NVIDIA Tesla T4 GPU.
All neural network methods are trained with batch size 64, for up to 100 epochs; early stopping occurs if loss does not improve in 5 epochs.
Each algorithm is trained on the train split, and the performance metrics are computed on the test split.
We refer to each combination of (algorithm, hyperparameter set, dataset) as an experiment .
By running 24 algorithms, most with up to 100 hyperparameters, on 85 datasets, this resulted in 84successful experiments, and by computing 315 metrics, our final meta-dataset of results includes more than 26 million evaluations.
We give a more detailed look at the breakdown of experiments in
Appendix A, and we discuss any potential biases in the resulting dataset in Section 4.
2.1 On the generalizability of rec-sys algorithms
If a rec-sys algorithm or set of hyperparameters performs well on one dataset and metric, will it perform well on other datasets or on other metrics?
Our first observation is that all algorithms perform well on some datasets, and poorly on others.
First we identify the best-performing hyperparameter set for each (algorithm, dataset) pair—to simulate hyperparameter optimization using our meta-dataset.
We then rank all algorithms for each dataset, according to several performance metrics.
If we focus on a single metric, then many algorithms are ranked first according to this metric on at least one dataset.
Take for example metric NDCG@1: 17 of 20 algorithms are ranked either first or second on at least one dataset.
The same is true for metric RECALL@50: all algorithms except for
SlopeOne, GlobalEffects, and Random are ranked either first or second on at least one dataset.
The same is true for many other metrics and cutoffs (see Table 9 in Appendix C).
Average performance is more varied: some algorithms tend to perform better than others.
Tableshows the mean, min (best) and max (worst) ranking of all 24 algorithms over all dataset and all
[Figure]
[Figure] [Figure]
P R E C @
[Figure]
C O V -
I
T E MH I T @
[Figure]
A m z - M o v
i e
T V
A m z - H
o m e I m p r v
A n i m e B oo k C r o ss
C i a o D V D E p i n i o n s F il m T r u s t J e s t e r M k t B i a sM o d
M v T w ee t M v l n sK
M v l n sM
M v l n s - H e t r e c
R e c i p
e s W
i k il e n s Y a h oo M v
[Figure]
accuracy and hit-rate metrics.
This includes metrics NDCG, precision, recall, Prec.-Rec.-Min-density, hit-rate, F1, MAP, MAP-Min-density, ARHR, and MRR (see Appendix A.6 for descriptions of these metrics).
Nearly all algorithms are ranked first for at least one metric on at least one dataset.
Many algorithms perform well on average.
Furthermore, most algorithms perform very poorly in some cases: the maximum rank is at least 14 (out of 20) for all algorithms.
To illustrate the changes in algorithm performance across datasets, Figure 2 shows the normalized metric values for eight algorithms across 17 dataset splits.
Some algorithms tend to perform well (Item-KNN and SLIM-BPR) and others poorly (Random, TopPop), but no algorithm clearly dominates for all metrics and datasets.
This is a primary motivation for our meta-learning pipeline descirbed in Section 3: different algorithms perform well for different datasets on different metrics, so it is important to identify appropriate algorithms for each setting.
Generalizability of hyperparameters.
While the previous section assessed the generalizability of pairs of (algorithm, hyperparameters), now we assess the generalizability of the hyperparameters themselves while keeping the algorithms fixed.
For a given rec-sys algorithm, we can tune it on a dataset i , and then evaluate the normalized performance of the tuned method on a dataset j , compared to the normalized performance of the best hyperparameters from dataset j .
In other words, we compute the performance of tuning a method on one dataset and deploying it on another.
In Figure 3, we run this experiment for all pairs of datasets (one dataset per dataset family).
We plot the hyperparameter transfer for three different algorithms, as well as the average over all algorithms which completed sufficiently many experiments across the set of hyperparameters.
For each given i , j , we create the set of hyperparameters that completed for the given algorithm on both datasets i and j , and then we use min-max scaling for the performance metric values of these hyperparameters on i and on j separately.
Therefore, all matrix values are between 0 and 1; a value close to 1 indicates that the best hyperparameters from dataset i are also nearly the best on dataset j .
A value close to 0 indicates that the best hyperparameters from dataset i are nearly the worst for dataset j .
Across
A m a z
o n A ll B e a u t y A n i m e
B oo k C r o ss i n g
C i a o D V D D a t i n g E p i n i o n s F il m T r u s t F r a pp e G o w a ll a J e s t e r 2 L a s t F M
M a r k e t B i a s A m a z o n
M a r k e t B i
a
s M o d C l o t h
M o v i e T w ee t
i n g s
M o v i e l e n s 20 M
N e t f li x P r
i z e
R e c i p
[Table]
s W
i k il e n s
Y a
[Table]
M
[Table]
i
[Table]
AmazonAllBeautyAnimeBookCrossingCiaoDVDDatingEpinionsFilmTrustFrappeGowallaJester2LastFMMarketBiasAmazonMarketBiasModClothMovieTweetingsMovielens20MNetflixPrizeRecipesWikilensYahooMovies
[Table]
m
[Table]
ll
[Table]
a u t y A n i m
[Table]
B oo k
[Table]
r
[Table]
ss i n g
C i a
[Table]
D V D D a t i n g E p i n i o n s F il m T r u s t F r a
pp e G
[Table]
w a ll a J e s t e r 2 L a s t F M
M a r k e t B i a s A m a z o n
M a r k e t B i
a
s M o d C l o t h
M o v i e T w ee t
i n g s
M o v i e l e n s 20 M
N
[Table]
t f li x P r i z e
R
[Table]
c i p
[Table]
n
[Table]
M
[Table]
i
[Table]
AmazonAllBeautyAnimeBookCrossingCiaoDVDDatingEpinionsFilmTrustFrappeGowallaJester2LastFMMarketBiasAmazonMarketBiasModClothMovieTweetingsMovielens20MNetflixPrizeRecipesWikilensYahooMovies
[Table]
C i a o D V D D a t i n g E p i n i o n s F il m T r u s t F r a pp e G o w a ll a J e s t e r 2 L a s t F M
M a r k e t B i a s A m a z o n
M a r k e t B i
a
s M o d C l o t h
M o v i e T w ee t
i n g s
M o v i e l e n s 20 M
N e t f li x P r
i z e
R e c i p
e s W
i k il e n s
Y a h oo M o v i e s
AmazonAllBeautyAnimeBookCrossingCiaoDVDDatingEpinionsFilmTrustFrappeGowallaJester2LastFMMarketBiasAmazonMarketBiasModClothMovieTweetingsMovielens20MNetflixPrizeRecipesWikilensYahooMovies
[Figure]
A m a z
o n A ll B e a u t y A n i m e
B oo k C r o ss i n g
C i a o D V D D a t i n g E p i n i o n s F il m T r u s t F r a pp e G o w a ll a J e s t e r 2 L a s t F M
M a r k e t B i a s A m a z o n
M a r k e t B i
a
s M o d C l o t h
M o v i e T w ee t
i n g s
M o v i e l e n s 20 M
N e t f li x P r
i z e
R e c i p
e s W
i k il e n s
Y a h oo M o v i e s
AmazonAllBeautyAnimeBookCrossingCiaoDVDDatingEpinionsFilmTrustFrappeGowallaJester2LastFMMarketBiasAmazonMarketBiasModClothMovieTweetingsMovielens20MNetflixPrizeRecipesWikilensYahooMovies
[Figure]
[Figure] [Figure]
well from i to j , while a value close to 0 indicates that the hyperparameters transfer poorly.
all algorithms, we see that it is particularly hard for hyperparameters to transfer to and from some datasets such as Gowalla and Jester2.
Furthermore, the majority of pairs of datasets do not have strong hyperparameter transfer.
Overall, these experiments give evidence that tuning the hyperparameters of an algorithm on one dataset and transferring to another dataset does not give high performance, motivating RecZilla which predicts the best hyperparameters for a given algorithm and dataset.
2.2 On the predictability of rec-sys algorithms
Can attributes of the rec-sys dataset be used to predict the performance of rec-sys algorithms? Dataset meta-features. We calculate 383 different meta-features to characterize each dataset.
These meta-features include statistics on the rating matrix—including basic statistics, the distributionbased features of Cunha et al.
[23], and landmark features [24]—which measure the performance of simple rec-sys algorithms on a subset of the training dataset.
Since these meta-features are used for algorithm selection, they are calculated using only the training split of each dataset.
For more details on the dataset meta-features, see Appendix A.4.
Algorithm performance prediction.
Table 2 shows the meta-features that are most highlycorrelated with the performance (PREC@10) of each algorithm, using their default parameters.
Abs. Correlation
[Table]
Several meta-features aare highly-correlated with algorithm performance; one of the simplest metrics— the mean of the item rating count distribution—is highly correlated with performance of two rec-sys algorithms.
This experiment motivates the design of RecZilla in the next section, which trains a model using dataset meta-features to predict the performance of algorithms on new datasets.
As a toy-model version of RecZilla, we train three different meta-learner functions (XGBoost, KNN, and linear regression) using our meta-dataset, to predict performance metric PREC@10 for 10 rec-sys algorithms with high average performance (see Appendix A for details).
We use leave-one-out evaluation for each meta-learner: one dataset family is held out for testing, while m are used for training.
Figure 4 shows the mean absolute error (MAE) of each meta-learner; these results are aggregated over 200 random samples of randomly-selected training dataset families.
MAE decreases as more dataset families are added, suggesting that it is possible to estimate rec-sys algorithm performance using dataset meta-features.
We also find that performance metrics are not the only values that can be predicted with dataset meta-features.
In particular, we find that the runtime of rec-sys algorithms is also highly correlated with different meta-features.
Furthermore, we compute a simple measure of dataset hardness , which we compute as, given a performance metric, the maximum value achieved for that dataset across all algorithms.
For example, if all 20 algorithms do not perform well on the MovieTweetings dataset, then we can expect that the MovieTweetings dataset is “hard”.
Once again, we find that certain dataset meta-features are highly correlated with dataset hardness.
For more details on meta-feature correlation with algorithm runtimes and dataset hardness, see Appendix A.
The fact that dataset meta-features are correlated with algorithm performance, algorithm runtimes, and dataset hardness is a strong positive signal that meta-learning is worthwhile and useful in the context of recommender systems.
We explore this direction further in the next section.
RecZilla: Automated Algorithm Selection
In the previous section, we found that (1) the best algorithm and hyperparameters strongly depend on the dataset and user-chosen performance metric, and (2) the performance of algorithms can be predicted from dataset meta-features.
Points (1) and (2) naturally motivate an algorithm selection approach to rec-sys powered by meta-learning.
In this section, we present RecZilla , which is motivated by a practical challenge: given a performance metric and a new rec-sys dataset, quickly identify an algorithm and hyperparameters that perform well on this dataset.
This challenge arises in many settings, e.g., when selecting good baseline algorithms for academic research, or when developing high-performing rec-sys algorithms for a commercial application.
We begin with an overview and then formally present our approach.
Overview.
RecZilla is an algorithm selection approach powered by meta-learning.
We use the results from the previous section as the meta-training dataset.
Given a user-specified performance metric, we train a meta-model that predicts the performance of each of a set of algorithms and hyperparameters on a dataset, by using meta-features of the dataset.
Given a new, unseen dataset, we compute the meta-features of the dataset, and then use the meta-model to predict the performance of each algorithm, returning the best algorithm according to the performance metric.
See Figure 1.
[Figure]
Preliminaries.
We start with notation for the rec-sys problem.
Let D denote a rec-sys dataset, consisting of a set of user-item interactions, split into a training and validation set.
Let a denote a rec-sys algorithm parameterized by a set of k ( a ) hyperparameters
ω ∈ H ( a ) ⊆
R k ( a ) , which is algorithm-dependent.
Suppose we train algorithm a on the training split of dataset D , using hyperparameters ω ; we denote the performance of algorithm a with hyperparameters ω on dataset D as ϕ ( a, ω , D ) ∈ R .
The function ϕ ( · , · , · ) represents a performance metric for the recommender system that is selected by the user; throughout this paper we refer to this function and its numerical value simply as performance .
In this paper, larger values of ϕ ( · , · , · ) indicates better performance.
We report the normalized performance defined as
[Equation]
where P maxD (resp.
P minD ) are the max (resp.
min) performance of any algorithm on D .
Next we define notation for the meta-learning problem.
Given a fixed performance function ϕ , let M =
( D i , a i , ω i , y i ) }
Mi =1 denote a meta- dataset consisting of M tuples.
Each tuple consists of a dataset D , algorithm a parameterized by ω , and performance y , where y i ≡ ϕ ( a i , ω i , D i ) .
Since many algorithms have a wide range of hyperparameters, we refer to the set of parameterized algorithms in the meta-dataset as S =
{ ( a i , ω i ) } Ni =1 .
We represent each dataset using vector d ∈ R m , where each element of d is a meta-feature of the dataset.
In this setting, the meta-learning task is to identify a function that estimates the performance of parameterized algorithm ( a, ω ) on dataset D .
3.1 The RecZilla Algorithm Selection Pipeline
The RecZilla algorithm selection pipeline takes as input a meta-dataset M built with a user-chosen performance function for the recommendation task ϕ ( · , · , · ) .
RecZilla can accommodate any performance function that is a computable function of the performance metrics present in the meta-dataset.
The RecZilla pipeline proposed here returns both an algorithm a ∈ A and a set of hyperparameters ω ∈ H ( a ) , so that RecZilla can be used without additional hyperparameter optimization.
Since the meta-learning problem is in the low-data regime, to guard against overfitting, we select a subset of the algorithms that have good coverage over the training dataset.
We similarly select only the meta-features which are most predictive of the user-selected performance metric for the selected algorithms.
Below we outline each of the steps used to build the proposed RecZilla pipeline.
Algorithm subset selection: We select a subset S ′ ⊆ S of n parameterized algorithms that collectively perform well on all datasets in the meta-training dataset M , according to performance function ϕ .
We aim to select an algorithm subset with high coverage over the set of known datasets, where coverage of a subset S ′ is defined as
[Equation]
In other words, the coverage of subset S ′ is the normalized performance metric of the best performing algorithm in S ′ , averaged over all datasets D .
Selecting a subset with maximum coverage is itself a difficult problem; we use a greedy heuristic as follows.
We begin with S ′
=
{} and iteratively add the parameterized algorithm ( a ∗ , ω ∗ ) ∈ arg max ( a, ω ) ∈S C ( S ′ ∪ { ( a, ω ) } ) to
S ′ until |S ′ | = n .
That is, we greedily ensure that the coverage is maximized at each step.
[Table]
Meta-feature selection: Similar to the previous point, we select a subset of meta-features with good coverage over the meta-training dataset M , where here coverage is defined in terms of the correlation between algorithm performance and each meta-feature (see Appendix A.4 for details).
Let M : D → R m denote the resulting function that maps a dataset to a vector of meta-features.
Meta-learning: We learn a function f : R m → R n , where f ( d ) =
ˆ y is a vector of the estimated performance metric of all parameterized algorithms in S on dataset meta-features d .
Note that the RecZilla pipeline has two hyperparameters: n , the number of parameterized algorithms in S ′ ; and m , the number of dataset meta-features used by the meta-learner.
In our experiments, we run an ablation study with both n and m , as well as different functions f for the meta-learning model.
Using RecZilla for Algorithm Selection.
After developing the RecZilla pipeline, we use the following steps to select an algorithm for a new dataset D ′ :
Calculate m meta-features of the dataset d ′ ← M
( D ′ ) .
Estimate the performance of all parameterized algorithms:
y ′
← f ( d ′ ) .
Return the parameterized algorithm in S ′ with the best estimated performance.
3.2 Experiments
In this section, we evaluate the end-to-end RecZilla pipeline.
We start by describing the specific versions of RecZilla used in our experiments.
We use four different meta-learning functions within
RecZilla: XGBoost [12], linear regression, k -nearest neighbors, and uniform random.
For KNN, we set k = 5 and use the L 2 distance from the selected meta-features.
Experimental setup.
Focusing on performance metric PREC@10, we build a meta-dataset M using all rec-sys datasets, algorithms, and meta-features described in Section 2.
We first use the algorithm selection and meta-feature selection procedures described above to select n = m =parameterized algorithms and meta-features.
For all experiments, we use the best 10 parameterized algorithms selected during this process.
We vary both the number of training meta-datapoints and meta-features; the datapoints and features are randomly selected over 50 random trials.
All RecZilla meta-learners are evaluated using leave-one-dataset-out evaluation: we iteratively select each dataset family as the meta-test dataset, and run the full RecZilla pipeline using the remaining datasets as the meta-training data.
Splitting on dataset families rather than datasets ensures that there is no test data leakage.
Then for each dataset D in the test set, we compare the performance metric of the predicted best parameterized algorithm ( a ′ , ω ′ ) to the performance metric of the ground-truth best algorithm y ∗ , using the percentage-difference-from-best: % Diff = 100 × ( y ∗ − ϕ
( D, a ′ , ω ′ ))
/y ∗ .
% Diff is between 0 and 100, and smaller values indicate better performance.
Results and discussion.
In Table 3, we compare RecZilla with two prior algorithm selection approaches: cunha2018 [27] and cf4c4-meta [25], which are a comprehensive depiction of all prior work (see Appendix B.4 for justification, and for details of the experiment).
Furthermore, note that cunha2018 has no open-source code, and cf4c4-meta only has code in R. We find that
RecZilla outperforms the other two approaches in both %Diff and in terms of the PREC@10 value of the rec-sys algorithm outputted by each meta-learning algorithm.
In Appendix C.3, Figure 5 (left) shows % Diff vs. the size of the meta-training set, and Figure 5 (right) shows the results of an ablation study on the number of selected meta-features m .
See
Appendix C.3 for more details and discussion.
Pre-trained RecZilla models.
We release pre-trained RecZilla models for PREC@10, NDCG@10, and Item-hit Coverage@10, trained with XGBoost on all 18 datasets, with algorithms n = 10 and meta-features m = 10 .
We also include a RecZilla model that predicts the Pareto-front of PREC@ and model training time, so that users can select their desired trade-off between performance and runtime.
Finally, we include a pipeline so that users can choose a metric from the list of 315 (or any desired combination of the 315 metrics) and train the resulting RecZilla model.
Conclusions, Limitations, and Broader Impact
In this work, we conducted the first large-scale study of rec-sys approaches: we compared 24 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics.
We showed that for a given performance metric, the best algorithm and hyperparameters highly depend on the dataset.
We also find that various meta-features of the datasets are predictive of algorithmic performance and runtimes.
Motivated by these findings, we created RecZilla, the first metric-independent, hyperparameter-aware algorithm selection approach to recommender systems.
Through empirical evaluation, we show that given a user-defined metric, RecZilla effectively predicts high-performing algorithms and hyperparameters for new, unseen datasets, substantially reducing the need for human involvement.
We not only release our code and pretrained RecZilla models, but we also release the raw experimental results so that users can train new RecZilla models on their own test metrics of interest.
This codebase, which includes a unified API for 85 datasets and 24 algorithms, may be of independent interest.
Limitations.
While our work progresses prior work along several axes, there are still avenues for improvement.
First, the meta-learning problem in RecZilla is low-data.
Although we added nearly all common rec-sys research datasets into RecZilla, the result is still only 85 meta-datapoints (datasets).
While we guarded against over-fitting to the training data in numerous ways, RecZilla can still be improved by more training data.
Therefore, as new recommender system datasets are released in the future, our hope is to add them to our API, so that RecZilla continuously improves over time.
Similarly, our hope is to add the most recent high-performing rec-sys approaches to our work, as well as algorithms released in the future.
This includes adding neural network-based approaches, in addition to the six that we have already included.
Another limitation is that RecZilla does not directly predict the performance of hyperparameters for algorithms on a given dataset.
Although care must be taken to not overfit, modifying RecZilla to predict the performance of an algorithm together with a set of hyperparameters is an interesting avenue for future work.
Finally, the magnitude of our evaluation ( 84 850 models trained) leaves our meta-data susceptible to biases based on experiment success/failures.
While we fixed many common errors such as out-of-memory errors, it was infeasible to give each experiment specific attention.
Therefore, RecZilla may have higher uncertainty for the datasets and algorithms that are more likely to fail.
An interesting future improvement to RecZilla would be to predict the likelihood that a new dataset will successfully train on a new dataset.
Broader impact.
Our work is “meta-research”: there is not one specific application that we target, but our work makes it substantially easier for researchers and practitioners to quickly train recommender system models when given a new dataset.
On the research side, this is a net positive because researchers can much more easily include baselines, comparisons, and run experiments on large numbers of datasets, all of which lead to more principled empirical comparisons.
On the applied side, our day-to-day lives are becoming more and more influenced by recommendations generated from machine learning models, which comes with pros and cons.
These recommendations connect users with needed items that they would have had to spend time searching for [54].
Although these recommendations may lead to harmful effects such as echo chambers [38, 55], techniques to identify and mitigate harms are improving [42, 69].
Acknowledgments and Disclosure of Funding
Dickerson was supported in part by NSF CAREER Award IIS-1846237, NIST MSE Award #20126334, DARPA GARD #HR00112020007, and DoD WHS Award #HQ003420F0035.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
[The main claims in the abstract and introduction reflect the paper’s contributions and scope.]
(b) Did you describe the limitations of your work?
[Yes] [See Section 4.]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] [See Section 4.]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] [We read the ethics review guidelines and ensured our paper conforms to them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
[We did not include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
[We did not include theoretical results.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] [We include the code, data, and instructions to reproduce the results here:
https: //github.com/naszilla/reczilla .]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] [See Section 2 and Appendix A.] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] [See Section 3.]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] [We include this information in Section 2].
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] [See Section 2 and Appendix A.] (b) Did you mention the license of the assets?
[N/A]
Our experiments were conducted only on publicly available datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We did not include new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
Our experiments were conducted only on publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
Our experiments were conducted only on publicly available datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
[We did not conduct research with human subjects.
] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
[We did not conduct research with human subjects.
] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[We did not conduct research with human subjects.]
A Experiment Details
This appendix outlines the algorithms, datasets, metrics, and hyperparameter selection used in
RecZilla, as well as the details of our procedure for generating the meta-dataset.
This codebase is publicly available 2 , and is written in Python.
The RecZilla codebase builds on another public Github repository 3 .
A.1 Generating Meta-Datasets
We generate the meta-dataset for reczilla using 24 rec-sys algorithms and 85 datasets.
We use a leave-one-out training/validation split for each dataset: for each user, the last interaction is held out for validation, and all remaining interactions are used for training.
For non-neural-network algorithms, each algorithm-dataset pair is given a 10 hour time limit for training and validation and trains/validates with up to 100 random hyperparameter sets (see Appendix A.5), on a single a n1-highmem-2 instance on Google Cloud (2 vCPUs, 13GB memory).
Neural-network based methods are trained using the default hyperparameters from their respective papers—meaning we do not sample different hyperparameter sets.
For each dataset, neural-network methods are run for up to 15 hours on an n1-highmem-16 Google Cloud instance (16 vCPUs, 104GB memory), with an NVIDIA Tesla T4 GPU.
All neural methods use batch size 64, and are trained for up toepochs; early stoppig occurrs if loss does not improve in 5 epochs.
During validation, we calculate 21 different performance metrics at 15 different cutoffs, for a total of 315 different metrics (see
Appendix A.6).
Out of all 1 530 dataset-algorithm combinations tested in our experiments, 1 404 of them completed the train/validation procedure with at least one hyperparameter set within thehour time limit.
Most failed experiments failed due to invalid hyperparameter values, and some failed due to memory errors.
Algorithm runtime varied substantially across algorithm family and dataset.
Table 4 shows runtime statistics over all experiments and all algorithms, for experiments that completed within the 10-hour time limit.
A.2 Rec-sys Algorithms Implemented in RecZilla
Our experiments use 24 rec-sys algorithms.
Algorithms with hyperparameters are associated with a hyperparameter space, as well as a set of “default” hyperparameters.
Table 5 lists each implemented algorithm, along with its hyperparameter space and default parameters.
In the current implementation, we define several versions of User-KNN and Item-KNN, with one version for each similarity metric.
This is done for convenience, since different KNN similarity metrics are associated with different hyperparameters.
However, in our experiment results we treat all versions of User-KNN and Item-KNN as the same algorithm.
All algorithms here use the interface from [28, 29] (their codebase is publicly available 4 ).
All but two of our 24 algorithms use the implementation from this codebase; two algorithms (CoClustering and
SlopeOne) use the implementation of Surprise [53], which is also publicly available.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
l
[Table] [Table]
between
[Table]
cosine
[Table]
between items is calculated
[Table]
the
[Table]
similarity
[Table]
Similarity between items is calculated using the Sørensen-Dice
[Table] [Table]
between items is calculated using the euclidean distance (l2 distance).
[Table]
Similarity between items is calculated using the
[Table]
index.
[Table]
ItemKNN
[Table]
ItemKNN-Tversky k-nearest neighbors, item-based.
[Table]
Similarity between items is calculated using the
Tversky index.
[Table]
factorization model that replaces user factors with the factors of items rated by that user.
Items have multiple corresponding factors.
[Table] [Table]
Computes the relevance between users and items based on random walks in a graph con- taining both users and items [20]
[Table]
Similar to
P3alpha, but uses a reweighing scheme to compensate for item popular- ity [
[Table] [Table]
A.3 RecZilla Datasets
The RecZilla codebase implements 88 datasets (3 additional datasets were added after our experiments on 85 datasets), derived from 20 dataset families.
All datasets are listed in Table
[Table]
[Table]
5.70EAmazonAppstoreforAndroid   7.80EAmazonArtsCraftsSewing 97, 112, 30, 2.80EAmazonAutomotive 300, 320, 100, 9.40EAmazonBaby 236, 64, 71, 5.10EAmazonBabyProducts 10, 9, 5, 2.10EAmazonBeauty 489, 249, 146, 1.30EAmazonBlues
4.40EAmazonBooks 11,498, 2,330, 1,686, 2.90EAmazonBuyaKindle 6, 1, 2, 1.30EAmazonCDsVinyl 1,705, 486, 245, 1.40EAmazonCellPhonesAccessories 319, 245, 7.51EAmazonChristian 1, 7,  3.60EAmazonClassical 2,  1.50EAmazonClothingShoesJewelry 1,615, 1,136, 496, 2.86EAmazonCollectiblesFineArt 1, 5,  8.12EAmazonComputers 4,  4.60EAmazonCountry 1,  1.90EAmazonDanceElectronic 4,  6.80EAmazonDavis   2.30EAmazonDigitalMusic 238, 266, 56, 1.60EAmazonElectronics 2,302, 476, 651, 7.40EAmazonFolk 2,  2.60EAmazonGiftCards   4.80E-
AmazonGospel 1,  1.40EAmazonGroceryGourmetFood 335, 166, 86, 2.30EAmazonHardRockMetal 1,  3.60EAmazonHealthPersonalCare 661, 252, 205, 1.30EAmazonHomeImprovement 3,  3.60EAmazonHomeKitchen 1,029, 410, 327, 7.70EAmazonIndustrialScientific 16, 45, 7, 4.75EAmazonInternational 5,  5.70EAmazonJazz 2,  1.50EAmazonKindleStore 1,387, 430, 213, 1.50EAmazonKitchenDining 3,  3.50EAmazonLatinMusic   2.60EAmazonLuxuryBeauty 1, 1,  1.20EAmazonMagazineSubscriptions 1, 1,  1.58EAmazonMiscellaneous 5,  4.80EAmazonMoviesTV 1,894, 200, 319, 3.00EAmazonMP3PlayersAccessories 1,  8.19EAmazonMusicalInstruments 92, 83, 29, 3.80EAmazonNewAge 1,  2.40EAmazonOfficeProducts 166, 130, 59, 2.10EAmazonOfficeSchoolSupplies 3,  6.05EAmazonPatioLawnGarden 134, 105, 54, 2.30EAmazonPetSupplies 291, 103, 93, 3.00EAmazonPop 5,  6.60EAmazonPurchaseCircles   4.70EAmazonRapHipHop   2.20EAmazonRB 2,  8.70EAmazonRock 4,  1.20EAmazonSoftware 29, 18, 9, 1.80EAmazonSportsOutdoors 751, 478, 238, 6.60EAmazonToolsHomeImprovement 751,440 412, 260, 132, 1.20EAmazonToysGames 549, 327, 164, 1.00EAmazonVideoGames 308, 50, 84, 7.30EAmazonWine 1,  2.10EAnime 7,669, 11, 69, 9.80EBookCrossing 323, 340, 22, 4.20ECiaoDVD 47, 16, 4, 6.20EDating 17,088, 168, 135, 7.50EEpinions 592, 139, 28, 1.50EFilmTrust 32, 2, 1, 1.20EFrappe 17, 4,  5.40EGoogleLocalReviews 4,867, 3,116, 818, 1.90EGowalla 3,735, 1,247, 91, 3.30EJester 1,640,  56, 2.10ELastFM 89, 17, 1, 2.70EMarketBiasAmazon 32, 9, 20, 1.70EMarketBiasModCloth 40, 1, 6, 5.80EMovielens100K 98, 1,  6.20EMovielens10M 9,833, 10, 69, 1.30EMovielens1M 986, 3, 6, 4.20EMovielens20M 19,723, 27, 138, 5.20EMovielensHetrec 851, 10, 2, 4.00EMovieTweetings 808, 38, 31, 6.70ENetflixPrize 99,521, 17, 476, 1.20ERecipes 819, 231, 35, 1.00EWikilens 26, 5,  1.90EYahooMovies 195, 11, 7, 2.15EYahooMusic 77,764, 98, 1,647, 4.81E-
[Table]
[Table]
A.4 Dataset Meta-Features & Meta-Feature Selection
We calculate a total of 383 meta-features for each rec-sys dataset, consisting of a few general meta-features, meta-features describing the distribution of ratings
[Table]
and performance of landmarkers.
For each dataset
[Table]
we extract the number of users, number of items, number of ratings
[Table]
and the ratio of items to users.
Furthermore, following the approach outlined in [23]
[Table]
we compute the sparsity of the matrix of interactions
[Table]
and we systematically obtain a series of meta-features based on different distributions that can be obtained by aggregating the ratings in several ways.
Distribution
meta-features The distribution meta-features are obtained in two steps, as described in [23].
First, we obtain a distribution.
We do this in one of seven ways.
We either take all of the ratings at once or we aggreggate ratings for either items or users in one of three different ways: sum, count, or mean.
For each of these seven distributions, we then compute ten different descriptive statistics: mean, maximum, minimum, standard deviation, median, mode, Gini index, skewness, kurtosis, and entropy.
This results in 70 distribution meta-features.
Landmarkers For landmarkers, we evaluate the performance of several baseline algorithms on a subset of the training set.
We first select the subsample.
Next, we partition this subsample into two sets: a “sub-training set” and a “sub-validation set”.
We train each landmarker on the sub-training set and compute performance metrics on the sub-validation set.
We compute the 19 performance metrics described in Section A.6, plus three more algorithm-independent metrics:
• Items in Evaluation Set: measures the fraction of items with at least one rating in the evaluation set.
This metric is algorithm-independent and only serves to describe the dataset and its split.
• Users in Evaluation Set: measures the fraction of users with at least one rating in the evaluation set.
This metric is algorithm-independent and only serves to describe the dataset and its split.
• Item Coverage: fraction of items that are ranked within the top K for at least one user.
All 22 metrics are evaluated at cutoffs 1 and 5, to create the meta-features.
The subsampling scheme is designed to satisfy several constraints.
We limit the number of users to 100 and the number of items to 250.
We also need to ensure there are at least 2 items rated per user so that the subsequent data split (holding out one item rating per user) does not result in cold users.
Furthermore, we ensure the number of items selected is at least 6 so that we can evaluate the performance metrics with the cutoff of 5.
We start the subsampling process by filtering by the users that have at least 2 ratings.
We next filter by the items all those users rated.
If this results in less than 6 items, we add a random sample of the remaining (cold) items back to the set to have 6 items in total.
Next, if the number of users is larger than 100, we take a random subsample of 100 of them and filter by those users.
As before, we filter by the items those users rated, and ensure this results in at least 6 items by adding random items back if needed.
If the number of items is still greater than 250, we must build an item subsample that results in at least two ratings per user.
For each user, we randomly choose two of the items the user rated, and we take the union of these item choices.
If this results in less than 250 items, we take a random sample of the remaining items to make the total number of items equal to 250.
Once this subsample is built, we split the subsample into the sub-training and sub-validation set by leaving 1 random item out for each user.
We are then ready to run our landmarkers on those sets.
Our landmarkers consist of TopPop, ItemKNN, UserKNN, and PureSVD.
For ItemKNN and
UserKNN, we use cosine similarity and set the number of neighbors k to 1 and 5.
For PureSVD, we use 1 and 5 for the number of latent factors.
This results in a total of 7 landmarkers.
Running all 7 landmarkers and computing all 22 metrics at the 2 different cutoffs results in a total of 308 landmarker meta-features.
A.5 Hyperparameter Sampling
For each algorithm with hyperparameters, we test up to 100 parameter sets, limited by the 10-hour time limit used in our experiments.
The first evaluated hyperparameter set is the default hyperparameters
The remaining 99 hyperparameters are sampled using the ranges specified in Table 5, using Sobol sampling.
A.6 Evaluation Metrics
Each of the evaluation metrics that we use during validation measures the quality of ranking based on the item ratings.
For each user, we generate predicted ratings for all items and rank the items according to the predicted rating (in descending order).
We trim these ranked lists at a given cutoff, which we denote by K .
We then compute different metrics using these user-wise top K items.
Some metrics also consider the set of relevant items within these top K , defined as those for which the user rated the item in the evaluation set.
We use different metrics, and we compute them using cutoffs in { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 15 , 20 , 30 , 40 , 50 } .
This results in 315 different metric/cutoff combinations total.
Only a small subset of these metrics are used in our analysis; however, any user-chosen metric (or combination of metrics) can be used to define a performance function for
RecZilla.
The metrics are calculated using the implementation in the public repository 8 that our codebase is built on.
Below is a list of metrics calculated during model evaluation:
• Average Popularity: measures the popularity of the recommended items.
The popularity for each item is the frequency with which it was rated in the training set.
These popularities
are normalized by the largest popularity.
Next, for each user, we compute the normalized popularity of each of the items within its top K and take their mean.
Finally, we average across all users.
• Average Reciprocal Hit-Rank (ARHR): similar to MRR, except the reciprocal ranks for all relevant items (not just the first) are summed together.
• Diversity (Gini):
computes the Gini diversity index 9 of the global distribution of items ranked within the top K across all users.
Higher values indicate higher diversity.
• Diversity (Herfindahl): computes the Herfindahl index [2] of the global distribution of items ranked within the top K across all users.
Higher values indicate higher diversity.
• Diversity (Shannon): computes the Shannon entropy of the global distribution of items ranked within the top K across all users.
• F1 Score: the harmonic mean between precision and recall.
• Hit Rate: fraction of users for which at least one relevant item is present within the top K .
• Item Coverage (Hit): fraction of relevant items that are ranked within the top K for at least one user.
• Mean Average Precision (mAP): the mean of the average precision across all users.
The average precision for a user is computed as follows: for any position i ≤ K occupied by a relevant item, we compute the precision at i .
We sum all of these precision values and divide the total by K .
• Mean Average Precision - Min Den: similar to mAP, but using a modified version of the average precision.
If the number of test items for the user is smaller than K , we divide the sum (in the last step of the average precision computation) by this number instead of by K .
• Mean Inter-List Diversity: measures how different the top K lists are for all users, as originally proposed in [86].
For each pair of users, we compute the fraction of items in their top K items that are not present in both lists.
Taking the average across all users yields the mean inter-list diversity.
The codebase implements a more efficient but equivalent way of computing this metric.
• Mean Reciprocal Rank (MRR): the mean of the reciprocal rank across all users.
The reciprocal rank for a user is the reciprocal of the rank of the most highly-ranked relevant item or 0 if there is none.
Normalized Discounted Cumulative Gain (NDCG): first, the discounted cumulative gain (DCG) of the top K ranking is computed by adding, for all relevant items in the top K , a gain discounted logarithmically in terms of the rank.
The NDCG is obtained by dividing the DCG of the ranking by that of an ideal ranking (a ranking ordered by relevance).
• Novelty: a metric that rewards recommending items that were not popular in the training set [86].
For each item, we compute the fraction of ratings in the training set that correspond to the item.
The novelty contributed by the item is computed by taking the negative logarithm of that fraction and dividing by the total number of items, so that items that were seldom seen in the training set result in high contributions.
Now, for any user, we compute the novelty as the sum of these contributions for the top K items.
Finally, we average the metric across all users.
• Precision: the fraction of items in the top K that are relevant, computed across all users.
• Precision Recall Min: similar to precision, but if there are less test items than K , the fraction is computed with respect to the number of test items.
• Recall: the fraction of relevant items that were placed within the top K , computed across all users.
• User Coverage: fraction of users both present in the evaluation set and for which the model is able to generate recommendations.
In practice, all the algorithms used were able to generate recommendations for all users, since our splitting procedures did not result in cold users, so this metric was always 1 for our datasets and algorithms.
[Table] [Table]
• User Coverage (Hit): fraction of users both present in the evaluation set and for which the model is able to generate at least one relevant recommendation within the top K items.
It is equal to the product of the hit rate and the user coverage.
Because the latter was always equal to 1, the user coverage (hit) was the same as the hit rate in our experiments.
A.7 Additional details and experiments from Section 2.
Recall that in Table
[Table]
we showed the meta-features that are most highly correlated with the performance (PREC@10) of each algorithm, using their default parameters.
In Table
[Table]
we run the same analysis using “training time” instead of PREC@10 as the metric.
We see that for some algorithms, the runtime is very highly correlated with certain meta-features such as “number of interactions”.
Next, we compute a simple measure of dataset hardness , which we compute as, given a performance metric, the negative of the maximum value achieved for that dataset across all algorithms.
For example, if all 20 algorithms do not perform well on the MovieTweetings dataset, then we can expect that the MovieTweetings dataset is “hard”.
In Table 8, we show the meta-features that are most highly correlated with the hardness of each algorithm, where hardness is calculated as -PREC@10.
We find that the entropy of the rating matrix is most correlated with dataset hardness.
Additional details from Section 2.2.
In this section, we give more details of the experimental setup of the experiment in Figure 4 from Section 2.2.
We compare three different meta-learner functions:
XGBoost, linear regression, and KNN with k =5 and L 2 distance.
In order to give a fair comparison, we use a fixed set of 10 high-performing rec-sys algorithms, and also a fixed set of 10 meta-features that have high correlation with the performance metric PREC@10.
The meta-learner models are trained to predict PREC@10 of all 10 algorithms.
We use leave-one-out evaluation for each meta-learner: one dataset family is held out for testing, while x other dataset families are used for training, for x from 2 to 20.
For each meta-learner, we compute the mean absolute error (MAE): the absolute differences between the prediction and ground truth for each algorithm are averaged.
These are then averaged over 200 trials of randomly-selected training dataset families.
B RecZilla Meta-Learning Pipeline
In this section, we give more details of the RecZilla pipeline, and we give an additional experiment in which we compare RecZilla to other existing rec-sys meta-learning approaches.
The RecZilla pipeline consists of the following components: initial algorithm selection, meta-feature selection, and finally the meta-learner for algorithm selection.
The purpose of the first two components is to reduce the dimensionality of the dataset and reduce the risk of overfitting for the classifier.
We describe each of these components in the following sections.
In all that follows, we assume that the user provides (a) a performance metric function ϕ , (b) the number of parameterized algorithms to be considered by the meta-learner n , and (c) the number of dataset meta-features to be considered by the meta-learner m .
In addition, we assume access to a meta-dataset M , such as the one described (and already pre-computed) in this paper.
B.1 Initial algorithm selection
We first select n parameterized algorithms which have high coverage over all datasets in meta-dataset
M (see Section 3.1 of the paper).
Since data is relatively scarce in this meta-learning task, we select a subset of n algorithms to reduce the dimensionality of the meta-learner prediction target.
B.2 Meta-feature selection
Since our meta-dataset M includes hundreds of features, we restrict our meta-learner to m features to avoid over-fitting.
This is the same approach taken by prior work [27].
It is computationally infeasible to find the set of m = 10 best meta-features out of a set of 383, since we would need to check (cid:0) 38310 (cid:1)
≈
10 19 combinations of meta-features.
Instead, we iteratively grow a set of m meta-features which are highly correlated with the user-specified performance metric, without selecting redundant features, by using a “greedy” approach (similar in spirit to prior work [27]).
Here we require that the (a) performance metric function ϕ is chosen ahead of time, and (b) a set of n parameterized algorithms have been selected.
We introduce some additional notation for this section to describe our metafeature selection process.
Let y i ∈ R |D| be the vector of performance metric ϕ for parameterized algorithm i ∈ 1 , . . . , n , for all datasets in D .
Let d j ∈ R |D| be the vector of meta-feature j for all datasets in D , and let J be the total number of meta-features.
Let F denote a set of feature indices that corresponds to selected features.
For each (algorithm, meta-feature pair), we first compute the absolute value of the Pearson correlation between the meta-feature and the performance of each parameterized algorithm i , across all datasets: c ij ← | corr ( y i , d j ) | for all i ∈ { 1 , . . . , n } = [ n ] and j ∈ { 1 , . . .
, J } =
[ J ] .
When computing this correlation, each sample is weighed inverse-proportionally to the size of the dataset family it corresponds to—to prevent large dataset families (such as Amazon) from dominating the correlation computation.
We select the first feature by finding the largest absolute correlation coefficient between any of the meta-features and parameterized algorithms, and we choose the meta-feature corresponding to it:
[Equation]
All remaining ( m − 1) meta-features are selected such that we maximize the improvement in the absolute correlation between the selected features and the selected algorithms’ performance.
This way, we avoid selecting highly correlated features.
Algorithm 1 gives a pseudocode description of this feature-selection process.
B.3 Metalearner for algorithm selection
The goal of the metalearner is to predict the performance of all n selected parameterized algorithms on a new dataset.
The input to this meta-learner is the set of m meta-features selected in the previous selection, and the output is an n -dimensional vector of performance metrics for all selected algorithms.
We treat this as a multi output regression problem, and our experiments test three different models: a
RegressiorChain with XGBoost as the base model, KNN with k = 5 and L 2 distance, and multinomial linear regression.
For training these meta-learner models, we used the squared error cost function.
To train the meta-learner we construct a final meta-dataset consisting of one tuple ( d , y ) for each dataset represented in M train , where d is a vector of m meta-features for the dataset, and y is a vector of the performance of n parameterized rec-sys algorithms in the set of selected parameterized algorithms S ′ .
Require: d j ∈ R |D| , ∀ j ∈ [ J ] ▷ J vectors of meta-features for each dataset
Require: y i ∈ R |D| , ∀ i ∈ [ n ] ▷ n vectors of performance metrics for each algorithm
Require: m >
▷ number of features to select
F ← {} ▷ indices of selected features x i = 0 , ∀ i ∈ [ n ] ▷ max abs.
correlation between any selected meta-feature and y i c ij ← | corr ( y i , d j ) | , ∀ i ∈ [ n ] , j ∈
[ J ] while | F | < m do j ′ ← arg max j ∈ [ J ] (cid:20) max i ∈ [ n ] ( c ij − x i ) (cid:21)
F ← F ∪ { j ′ } x i ← max { x i , c ij ′ } , ∀ i ∈ [ n ] ▷ update the max.
abs.
correlation for each alg.
end while return F
B.4 Additional details on comparisons to other algorithm selection methods for rec-sys
As described in section of Section 1, there are a few existing works on algorithm selection for recommender systems.
In this section, we describe the two most relevant and recent works in more detail, and we give more details on the experiment in Section 3 that empirically compared RecZilla with existing works (Table 3).
Multiple approaches for algorithm selection for recommender systems were developed betweenand 2018 [3, 23, 35, 43, 52], but in 2018, Cunha et al.
[27] gave a thorough empirical analysis of all recommender systems studied up to that point.
They showed that the best approach was to use meta-feature selection over 74 distribution-based meta-features, and to use polynomial SVM as the meta-model.
Their study used 38 total datasets.
We refer to this algorithm as cunha2018 .
Two more algorithm selection for rec-sys approaches were released after Cunha et al.
CF4CF [26], which used subsampling landmarkers, and CF4CF-META, which combined CF4CF with the earlier meta-learning approaches described in the previous paragraph.
Specifically, CF4CF-META uses landmarkers as well as the 74 distribution-based meta-features.
It also uses KNN as the meta-model.
Their study uses 38 total datasets.
We refer to this approach as cf4cf-meta .
Given that cunha2018 subsumes all work prior to 2018, and cf4c4-meta subsumes all subsequent work, we compare RecZilla against these two algorithms as a comprehensive depiction of all prior work.
Note that cunha2018 has no open-source code, and cf4c4-meta only has code in R. Furthermore, in order to give a more fair empirical study, we implement both approaches directly within our codebase.
Additionally, to give a fair comparison, each model uses the same meta-training datasets, algorithm selection procedure, and base algorithms.
Since a main novelty of RecZilla is predicting hyperparameters as well as algorithms, the other two approaches are only given the algorithms with the default hyperparameters.
We run an experiment with cunha2018 , cf4c4-meta , and RecZilla, in the same setting as Section 3.2 (i.e., the setting of Figure 5), where we run leave-one-dataset-out evaluation and average the results over all test datasets.
The algorithms are given all 18 dataset families not in the test set, to use as training data.
We run 50 trials for all 19 possible test sets in the leave-one-dataset-out evaluation, for a total of 950 trials.
See Table 3.
RecZilla outperforms the other two approaches in both %Diff and in terms of the PREC@10 value of the rec-sys algorithm outputted by each meta-learning algorithm.
C Additional Results and Discussion C.1 Detailed algorithmic results
Tables 9, 10, and 11 show the min (best), max (worst) and mean rank for each algorithm, over all 85 datasets, over a subset of metrics.
Most algorithms are ranked in the top-3 for each metric, on at least one dataset; furthermore, most algorithms are ranked poorly on at least one dataset.
However, average performance varies widely across algorithms; for example, Item-KNN is has an average rank at most 3 for most metrics, while TopPop, GLobalEffects, SlopeOne, have average ranks closer to 20.
C.2 A guide to practitioners
In this section, we lay out the key takeaways and insights for practical recommender system design, so that practitioners can get the largest value from our work and use our work most effectively.
Insights from Section 2.
In Section 2, we presented a large-scale empirical study of rec-sys algorithms, studying the generalizability and predictability of rec-sys algorithms.
The key takeaway from this section for practitioners is that rec-sys algorithms are not generalizable (see Table 1 and
Figure 2).
Therefore, practitioners who have used an algorithm successfully on one rec-sys dataset will likely need to try other diverse types of algorithms when starting on a brand new dataset, even if using a neural network-based algorithm.
Practitioners may also use our analysis for intuition when working with RecZilla models, in a number of different ways.
The particular use cases and goals of a practitioner may be very specific, but we give a few examples on how to leverage insights from Section 2 below.
• Table 8 gives properties of a dataset that are predictive of “dataset hardness.”
Of course, this is based on correlation alone, but it suggests that a dataset is “harder” than average if the entropy of ratings is greater than 2.07 (this value can be computed directly inside our codebase 10 ).
If less than 2.07, the practitioner can continue as usual, but if greater than 2.07, it may be desirable to check if the dataset is noisy and can be cleaned, or if more data can be obtained.
Or, it could be a sign that the practitioner should expect to spend more time than usual tuning the recommender system model (after running RecZilla, of course).
• If the practitioner is concerned with the training time and latency of their recommender system model, then they can estimate the runtimes in advance.
For example, according to Table 4, number of interactions has a very high correlation with the training time of MF-FunkSVD .
By running a simple regression, we have the following formula: runtime of MF-FunkSVD = 0 .
· ( number of interactions ) + 0 .
Similar formulas can be computed for other algorithms, by using our codebase 11 .
• If the practitioner is interested in using a specific algorithm on a variety of datasets, then they can gain insights by looking up dataset meta-features that specifically have high correlation with the performance metric and runtime of that algorithm.
For example, a practitioner might prefer to use the CoClustering algorithm for the interpretable clusters that it gives.
By looking at Table 2, and by computing additional correlations using our codebase 11 , the practitioner can find the insight that Median of item rating count distribution and Median of item rating sum distribution are both predictive of the performance of CoClustering, which may be useful to know as a sanity check when tuning and deploying CoClustering on different datasets.
A guide for using RecZilla.
The goal of RecZilla is to allow practitioners to very quickly train a model that performs well, when faced with a new dataset.
The practitioner may then choose to further tune the model to reach even stronger performance.
The first step for the practitioner is to define their objective.
If the objective is PREC@10, NDCG@10, or Item-hit coverage@10, then the practitioner can use one of the pre-trained models included in our repository.
First, follow the installation instructions in the README.
11 Next, follow the instructions in the Main script overview section of the README.
Specifically, choose the metamodel_filepath that matches the desired objective.
If the desired objective is not one of the above three metrics, then the practitioner shoudl train a new meta-model.
This can be done by following the instructions in the Training a new meta-model
T a b l e 9 :
T h e p e rf
o r m a n ce
o f eac
h r ec - s y s a l g o r it h m v a r i e s d e p e nd
i ng on t h e
d a t a s e t a nd e v a l u a ti on m e t r i c .
E ac h r o w c
o
rr e s pond s t
o a d
i ff e r e n t m e t r i c ( ND
C G , p r ec i s i on , r ec a ll , a nd h
it -r a t e ) a t a p a r ti c u l a r c u t o ff .
E ac h a l go r
it
h m ( c o l u m n s ) a r e r a nk e d acc
o r d
i ng t
o eac
h m e t r i c , v a l u e s s ho w t
h e “ m i n / m a x ( m ea n ) ” r a nk ov e r a ll d a t a s e t s f o r w h
i c h a t l ea s t 10 a l go r
it h m s p r od u ce d a r e s u lt .
T h
i s t a b
l
e
i n c l ud e s a ll 24 i m p l e m e n t e
d a l go r it h m s .
I n t h i s t a b l e , m a x
i m u m r
a nk i s 24 , a nd l o w e r r a
nk i nd
i ca t e s b e tt e r p e rf
o r m a n ce .
T h i s t a b
l
e
i s c o n ti nu e d on t h e n e x t p a g e .
( P a r t 1 o f 3 )
I t e mK N N
U s e
r- K
N N
R P 3b e t a R P 3b e t a i
A L S
E A S ER
S l op e
O n e
C o C l u s t e r i ng
ND
C G @1 /( 1 . 5 ) 2 /( 11 .
ND
C G @1 /( 1 . 8 ) 2 /( 9 . 3 ) 2 /( 6 . 0 ) 1 /( 4 . 0 ) 1 /( 7 . 5 ) 1 /( 4 . 8 ) 9 /( 16 .
ND
C G @1 /( 2 . 2 ) 1 /( 7 . 4 ) 2 /( 5 . 9 ) 1 /( 4 . 0 ) 2 /( 7 . 4 ) 1 /( 5 . 2 ) 12 /( 17 .
ND
C G @1 /( 2 . 3 ) 1 /( 6 . 6 ) 2 /( 6 . 2 ) 1 /( 4 . 3 ) 1 /( 7 . 2 ) 1 /( 5 . 2 ) 14 /( 18 .
ND
C G @1 /( 2 . 9 ) 1 /( 5 . 9 ) 2 /( 6 . 4 ) 1 /( 4 . 9 ) 1 /( 6 . 5 ) 1 /( 5 . 8 ) 13 /( 18 .
P R E
C I S
I
O N @1 /( 1 . 5 ) 2 /( 11 .
P R E
C I S
I
O N @1 /( 1 . 9 ) 2 /( 8 . 4 ) 1 /( 5 . 9 ) 1 /( 3 . 9 ) 1 /( 7 . 4 ) 1 /( 4 . 6 )
P R E
C I S
I
O N @1 /( 2 . 6 ) 1 /( 6 . 1 ) 1 /( 5 . 9 ) 1 /( 4 . 2 ) 1 /( 7 . 1 ) 1 /( 5 . 6 ) 13 /( 17 .
P R E
C I S
I
O N @1 /( 2 . 9 ) 1 /( 4 . 9 ) 1 /( 6 . 6 ) 1 /( 4 . 6 ) 1 /( 6 . 2 ) 1 /( 6 . 6 ) 14 /( 18 .
P R E
C I S
I
O N @1 /( 4 . 3 ) 1 /( 4 . 1 ) 1 /( 7 . 3 ) 1 /( 5 . 8 ) 1 /( 5 . 4 ) 2 /( 8 . 1 ) 13 /( 18 .
R E C
A LL @1 /( 1 . 5 ) 2 /( 11 .
R E C
A LL @1 /( 1 . 9 ) 2 /( 8 . 4 ) 1 /( 5 . 9 ) 1 /( 3 . 9 ) 1 /( 7 . 4 ) 1 /( 4 . 6 ) 7 /( 16 .
R E C
A LL @1 /( 2 . 6 ) 1 /( 6 . 1 ) 1 /( 5 . 9 ) 1 /( 4 . 2 ) 1 /( 7 . 1 ) 1 /( 5 . 6 ) 13 /( 17 .
R E C
A LL @1 /( 2 . 9 ) 1 /( 4 . 9 ) 1 /( 6 . 6 ) 1 /( 4 . 6 ) 1 /( 6 . 2 ) 1 /( 6 . 6 ) 14 /( 18 .
R E C
A LL @1 /( 4 . 3 ) 1 /( 4 . 1 ) 1 /( 7 . 3 ) 1 /( 5 . 8 ) 1 /( 5 . 4 ) 2 /( 8 . 1 ) 13 /( 18 .
H I TR
A
TE @1 /( 1 . 5 ) 2 /( 11 .
H I TR
A
TE @1 /( 1 . 9 ) 2 /( 8 . 4 ) 1 /( 5 . 9 ) 1 /( 3 . 9 ) 1 /( 7 . 4 ) 1 /( 4 . 6 ) 7 /( 16 .
H I TR
A
TE @1 /( 2 . 6 ) 1 /( 6 . 1 ) 1 /( 5 . 9 ) 1 /( 4 . 2 ) 1 /( 7 . 1 ) 1 /( 5 . 6 ) 13 /( 17 .
H I TR
A
TE @1 /( 2 . 9 ) 1 /( 4 . 9 ) 1 /( 6 . 6 ) 1 /( 4 . 6 ) 1 /( 6 . 2 ) 1 /( 6 . 6 ) 14 /( 18 .
H I TR
A
TE @1 /( 4 . 3 ) 1 /( 4 . 1 ) 1 /( 7 . 3 ) 1 /( 5 . 8 ) 1 /( 5 . 4 ) 2 /( 8 . 1 ) 13 /( 18 .
T a b l e 10 :
C on ti nu a ti on o f
T a b l e 9 ( P a r to f 3 ) .
M F - A s y S V D
M F - F unk S V
D
S V D
N M F
M F - B P
R
S L I M - B P R
S L I M - E l a s ti c N e t
ND
C G @2 /( 10 .
ND
C G @2 /( 11 .
ND
C G @4 /( 11 .
ND
C G @4 /( 11 .
ND
C G @5 /( 11 .
P R E
C I S
I
O N @2 /( 10 .
P R E
C I S
I
O N @1 /( 10 .
P R E
C I S
I
O N @1 /( 10 .
P R E
C I S
I
O N @1 /( 10 .
P R E
C I S
I
O N @1 /( 9 . 7 ) 1 /( 9 . 0 ) 1 /( 5 . 4 ) 1 /( 6 . 8 ) 1 /( 11 .
R E C
A LL @2 /( 10 .
R E C
A LL @1 /( 10 .
R E C
A LL @1 /( 10 .
R E C
A LL @1 /( 10 .
R E C
A LL @1 /( 9 . 7 ) 1 /( 9 . 0 ) 1 /( 5 . 4 ) 1 /( 6 . 8 ) 1 /( 11 .
H I TR
A
TE @2 /( 10 .
H I TR
A
TE @1 /( 10 .
H I TR
A
TE @1 /( 10 .
H I TR
A
TE @1 /( 10 .
H I TR
A
TE @1 /( 9 . 7 ) 1 /( 9 . 0 ) 1 /( 5 . 4 ) 1 /( 6 . 8 ) 1 /( 11 .
T a b l e 11 :
C on ti nu a ti on o f
T a b l e 9 ( P a r to f 3 ) .
I- n e u r a l
U - n e u r a l
S p ec t r a l - C F
D EL FM L P
D EL FE F
M u lt - V
A E
G l ob a l
E ff ec t s
T op P op
R a ndo m
ND
C G @13 /( 15 .
ND
C G @13 /( 15 .
ND
C G @12 /( 15 .
ND
C G @11 /( 15 .
ND
C G @13 /( 15 .
P R E
C I S
I
O N @13 /( 15 .
P R E
C I S
I
O N @10 /( 15 .
P R E
C I S
I
O N @12 /( 15 .
P R E
C I S
I
O N @12 /( 15 .
P R E
C I S
I
O N @13 /( 16 .
R E C
A LL @13 /( 15 .
R E C
A LL @10 /( 15 .
R E C
A LL @12 /( 15 .
R E C
A LL @12 /( 15 .
R E C
A LL @13 /( 16 .
H I TR
A
TE @13 /( 15 .
H I TR
A
TE @10 /( 15 .
H I TR
A
TE @12 /( 15 .
H I TR
A
TE @12 /( 15 .
H I TR
A
TE @13 /( 16 .
[Figure]
section of the README.
In particular, any one of the 315 metrics can be used as the objective (or any computable function of these metrics).
Once the meta-model is trained, then the above instructions can be used.
Final takeaways.
Our work gives both intuition for working with rec-sys models, as well as a meta-learning pipeline that substantially lowers the level of human involvement when designing a high-performing rec-sys model.
Specifically, practitioners can use our large-scale empirical study to gain insights such as how generalizable a given algorithm is, and how “hard” it will be to train a high-performing model, compared to other datasets.
On the other hand, practitioners can follow the steps in our README to quickly train a high-performing model when faced with a brand new dataset.
C.3 Additional experiments from Section
In this section, we give more experiments with Reczilla.
Figure 5 (left) displays % Diff vs. the size of the meta-training set, and Figure 5 (right) displays the results of an ablation study on the number of selected meta-features m .
All results are averaged over all leave-one-out folds and 50 random trials.
Generally, XGBoost and KNN outperform the linear model and the random baseline, with XGBoost achieving top performance when using 10 meta-features and the maximum number of training datasets.
Furthermore, the number of datasets in the meta-training set matters more than the metalearning model itself.
For example, the improvement of XGBoost from 4 to 10 and to 18 training datasets is larger than the difference in performance between XGBoost and KNN at 4 and 10 training datasets, respectively.
Finally, we find that the optimal number of meta-features for XGBoost and
KNN peaks between 10 and 40.
C.4 Meta-learner results with additional metrics
The results from Figures 4 and 5 were generated using PREC@10 as the base recommender system metric.
Now, we re-run these experiments using COVERAGE@50 and HIT-RATE@5 as the base metrics, to show variety.
See Figure 6.
[Figure]