On the Generalizability and Predictability of Recommender Systems
Anonymous Author(s)
AfﬁliationAddress email
Abstract
While other areas of machine learning have seen more and more automation,designing a high-performing recommender system still requires a high level ofhuman effort.
Furthermore, recent work has shown that modern recommendersystem algorithms do not always improve over well-tuned baselines.
A naturalfollow-up question is, “how do we choose the right algorithm for a new datasetand performance metric?”
In this work, we start by giving the ﬁrst large-scalestudy of recommender system approaches by comparing 24 algorithms and 100sets of hyperparameters across 85 datasets and 315 metrics.
We ﬁnd that thebest algorithms and hyperparameters are highly dependent on the dataset andperformance metric, however, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets.
Motivatedby these ﬁndings, we create RecZilla, a meta-learning approach to recommendersystems that uses a model to predict the best algorithm and hyperparameters fornew, unseen datasets.
By using far more meta-training data than prior work,
RecZilla is able to substantially reduce the level of human involvement whenfaced with a new recommender system application.
We not only release our codeand pretrained RecZilla models, but also all of our raw experimental results, sothat practitioners can train a RecZilla model for their desired performance metric:https://anonymous.4open.science/r/anon-reczilla-51FC .
Introduction
Due to the large computational resources for training machine learning models, researchers havefound many ways to repurpose existing computation.
For example, it is common to start with apretrained ImageNet classiﬁcation model for computer vision tasks [29, 73, 48], or a pretrained BERTmodel for natural language tasks [30, 63].
These approaches work well because the core problems arelargely homogeneous; for example, any computer vision model at its core must be able to distinguishedges, colors and shapes.
Even a task-speciﬁc architecture can be found automatically through neuralarchitecture search [34], since the building blocks such as convolutional layers stay the same.
On the other hand, recommender system (rec-sys) research has followed a different trajectory: despitetheir widespread usage across many e-commerce, social media, and entertainment companies such as
Amazon, YouTube, and Netﬁx [20, 39, 75], there is far less work in reusing models or automating theprocess of selecting models.
Many rec-sys techniques are designed and optimized with just a singledataset in mind [39, 46, 20, 56, 79].
Intuitively, this might be because each rec-sys application ishighly unique based on the dataset and the target metric.
For example, a typical user session looksvery different among e.g. YouTube, Home Depot, and AirBnB [20, 56, 46].
However, this intuitionhas not been formally established.
Furthermore, recent work has shown that neural recommender system algorithms do not always improve over well-tuned baselines such as k -nearest neighbor and
[Table]
RecZilla pipeline is built using a meta-dataset M that includes many different performance metrics evaluated on many different rec-sys algorihtms on many different datasets; we estimate algorithm performance using dataset meta-features.
To avoid over-ﬁtting and reduce runtime, users can limit the number of algorithms and meta-features to constants n and m .
matrix factorization [27].
A natural question is then, “how do we choose the right algorithm for anew dataset and performance metric?”
In this work, we show that the best algorithm and hyperparameters are highly dependent on thedataset and user-deﬁned performance metric.
Speciﬁcally, we run the ﬁrst large-scale study ofrec-sys approaches by comparing 24 algorithms across 85 datasets and 315 metrics.
For each datasetand algorithm pair, we test up to 100 hyperparameters (given a 10 hour time limit per pair).
Thecodebase that we release, which includes a uniﬁed API for a large, diverse set of algorithms, datasets,and metrics, may be of independent interest.
We show that the algorithms do not generalize – theset of algorithms which perform well changes substantially across dataset and across performancemetrics.
Furthermore, the best hyperparameters of a rec-sys algorithm on one dataset often performsigniﬁcantly worse than the best hyperparameters on a different dataset.
Although we show that thereare no universal algorithms that work well on most datasets, we do show that various meta-featuresof the dataset can be used to predict the performance of rec-sys algorithms.
In fact, the same metafeatures are also predictive of the runtime of rec-sys algorithms as well as the “dataset hardness” –how challenging it is to ﬁnd a high-performing model on a particular dataset.
Motivated by these ﬁndings, we introduce RecZilla, a meta-learning-based algorithm selectionapproach (see Figure 1) inspired by SATzilla [81].
At the core of RecZilla is a model that, given auser-deﬁned performance metric, predicts the best rec-sys algorithm and hyperparameters for a newdataset based on meta dataset features such as number of users and items, and spectral properties ofthe interaction matrix.
We show that RecZilla quickly ﬁnds high-performing algorithms on datasets ithas never seen before.
While there has been prior work on meta-learning for recommender systems[25, 26], no prior work is metric-independent, searches for hyperparameters as well as algorithms,or considers more than nine dataset families.
By running an ablation study on the number of metatraining datasets, we show that more dataset families are crucial to the success of RecZilla.
We releaseready-to-use, pretrained RecZilla models for common test metrics, and we release the raw resultsfrom our large-scale study, along with code so that practitioners can easily train a new RecZilla modelfor their speciﬁc performance metric of interest.
Our contributions.
We summarize our main contributions below
• We run a large-scale study of recommender systems, showing that the best algorithm and hyperparameters are highly dependent on the dataset and user-deﬁned performance metric.
We also
show that dataset meta-features are predictive of the performance of algorithms.•
We create RecZilla, a meta-learning-based algorithm selection approach which, given a perfor-
mance metric, efﬁciently predicts the best algorithm and set of hyperparameters on new datasets.
• We release a public repository containing 85 datasets and 24 rec-sys algorithms, accessed througha uniﬁed API.
Furthermore, we release both pretrained RecZilla models, and raw data so thatusers can train a new RecZilla model on their desired metric.
Related Work Recommender systems are a widely studied area of research [10].
Common approaches include k -nearest neighbors [1], matrix factorization [55, 61], and deep learning approaches[20, 39, 75].
For a survey on recommender systems, see [10, 4].
A recent meta-study showedthat of the 12 published neural rec-sys approaches published at top conferences between 2015 and2018, 11 performed worse than well-tuned baselines (e.g. nearest neighbor search or linear models)[27].
Another recent paper found that the relative performance of rec-sys algorithms can changesigniﬁcantly based on the choice of datasets used [14].
Algorithm selection for recommender systems was ﬁrst studied in 2011 [50] by using a graphrepresentation of item ratings.
Follow-up work used dataset meta-features to select the best nearestneighbor and matrix factorization algorithms [33, 3, 41].
Subsequent work focused on improvingthe model and framework [26] including studying 74 meta-features systematically [22].
More recentapproaches from 2018 run meta-learning for recommender systems by casting the meta-problem itselfas a collaborative ﬁltering problem.
Performance is then estimated with subsampling landmarkers[23, 25, 24].
No prior work in algorithm selection for rec-sys includes open-source Python code.
There is also work on automated machine learning (AutoML) for recommender systems, withoutmeta-learning [80, 6, 44, 45].
Finally, we note that meta-learning across rec-sys datasets is not to beconfused with the body of work on meta-learning user preferences within a single rec-sys dataset[16, 17, 18].
To the best of our knowledge, no meta-learning or AutoML rec-sys paper has runexperiments on more than nine dataset families or four test metrics, and no prior work predictshyperparameters in addition to algorithms.
Analysis of Recommender Systems
In this section, we present a large-scale empirical study of rec-sys algorithms across a large, diverseset of datasets and metrics.
We assess the following two research questions.
Generalizability.
If a rec-sys algorithm or set of hyperparameters performs well on one datasetand metric, will it perform well on other datasets or on other metrics?
Predictability.
Given a metric, can various dataset meta-features be used to predict the performance of rec-sys algorithms?
Algorithms, datasets, and metrics implemented.
We present full results for 20 rec-sys algorithms,including methods from recent literature and common baselines.
Methods include ﬁve similarityand clustering-based methods: User-KNN [72], Item-KNN [74], P3Alpha [19], RP3Beta [70], and
Co-Clustering [37]; six Matrix-Factorization (MF) methods: MF-FunkSVD, MF-AsySVD [54],
MF-BPR [71], IALS [49], Pure-SVD, Non-negative matrix factorization (NMF) [21]; ﬁve methodsbased on linear models: Global-Effects, SLIM-BPR [7], SLIM-Elastic-Net [60], EASE-R [76], and
SlopeOne [59]; two simple baselines: Random and Top-Pop; and two neural network based methods:
User-NeuRec [82] and Mult-VAE [62].
We also include partial results (on 8-10 datasets each) forfour more neural network based methods: DELF-EF [13], DELF-MLP [13], Item-NeuRec [82], and
Spectral-CF [83] (with results included in Tables 9, 10, and 11).
These algorithms were chosen dueto their high performance, popularity, and speed.
For many algorithms, we used the implementationsfrom the codebase of Dacrema et al.
For full details of the algorithms, see Appendix A.
We run the algorithms on 85 datasets from 19 dataset “families”:
Amazon [69], Anime 1 , BookCrossing [85], CiaoDVD [57, 43], Dating (Libimseti.cz) [57, 58], Epinions [66, 65], FilmTrust [42],
Frappe [8], Gowalla [15], Jester2 [38], LastFM [11], MarketBias-Electronics and MarketBiasModCloth [78], MovieTweetings [32], Movielens [47], NetﬂixPrize [9], Recipes [64], Wikilens [35],and Yahoo 2 .
Here, a “dataset family” refers to an original dataset, while “dataset” refers to asingle train-test split drawn from the original dataset, which may be a small subset of the original.
We implemented the majority of rec-sys datasets commonly used for research; to the best of ourknowledge, this is the largest number of rec-sys datasets accessible in a single open-source repository.
[Table]
We use 23 different “base” metrics: ARHR, Average Popularity, Diversity Similarity, F1 Score, Gini
[Table]
Index, Herﬁndahl Index, Hit Rate, Item Coverage, Item-hit Coverage, Precision (PREC), Precision-
[Table]
Recall Min Denominator, Recall, MAP, MAP Min Denominator, Mean Inter-List Diversity, MRR,
NDCG, Novelty, Shannon Entropy, User Coverage, User-hit Coverage.
All of these metrics arecomputed at cutoffs { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 15 , 20 , 30 , 40 , 50 } , for a total of 315 different metrics.
See Appendix A.6 for more details.
These metrics include the most popular from the literature, andwe use the Dacrema et al.
[27] implementations.
Experimental design.
Each dataset’s train, validation, and test split is based on leave-last-
k -out(and our repository also includes splits based on global timestamp).
We use a random hyperparametersearch for all methods, with the exception of neural network based methods.
Since neural networksrequire far more resources to train (longer training time, and requiring GPUs), we use only thedefault hyperparameters for neural algorithms.
For each non-neural algorithm, we expose severalhyperparameters and give ranges based on common values.
For each dataset, we run each algorithmon a random sample of up to 100 hyperparameter sets.
Each algorithm is allocated a 10 hour limitfor each dataset split; we train and test the algorithm with at most 100 hyperparameter sets on ann1-highmem-
Google Cloud instance, until the time limit is reached.
Each neural network methodis trained on each dataset using the default hyperparameters used in its respective paper, with a timelimit of 15 hours on an NVIDIA Tesla T4 GPU.
All neural-network methods are trained with batchsize 64, for up to 100 epochs; early stopping occurs if loss does not improve in 5 epochs.
Each algorithm is trained on the train split, and the performance metrics are computed on the testsplit.
We refer to each combination of (algorithm, hyperparameter set, dataset) as an experiment .
Byrunning 24 algorithms, most with up to 100 hyperparameters, on 85 datasets, this resulted in 84 850successful experiments, and by computing 315 metrics, our ﬁnal meta-dataset of results includesmore than 26 million evaluations.
We give a more detailed look at the breakdown of experiments in
Appendix A, and we discuss any potential biases in the resulting dataset in Section 4.
[Equation]
On
If a rec-sys algorithm or set of hyperparameters performs well on one dataset and metric, will itperform well on other datasets or on other metrics?
Our ﬁrst observation is that all algorithms perform well on some datasets, and poorly on others.
Firstwe identify the best-performing hyperparameter set for each (algorithm, dataset) pair—to simulatehyperparameter optimization using our meta-dataset.
We then rank all algorithms for each dataset,according to several performance metrics.
If we focus on a single metric, then many algorithms are ranked ﬁrst according to this metric on atleast one dataset.
Take for example metric NDCG@1: 17 of 20 algorithms are ranked either ﬁrst orsecond on at least one dataset (the exceptions are Mult-VAE, SlopeOne, and Random).
The sameis true for metric RECALL@50: all algorithms except for SlopeOne, GlobalEffects, and Randomare ranked either ﬁrst or second on at least one dataset.
The same is true for many other metrics andcutoffs (see Table 9 in Appendix C).
Average performance is more varied: some algorithms tend to perform better than others.
Table 1shows the mean, min (best) and max (worst) ranking of all 24 algorithms over all dataset and allaccuracy and hit-rate metrics.
This includes metrics NDCG, precision, recall, Prec.-Rec.-Min-density,hit-rate, F1, MAP, MAP-Min-density, ARHR, and MRR (see Appendix A.6 for descriptions of thesemetrics).
Nearly all algorithms are ranked ﬁrst for at least one metric, on at least one dataset; twoexceptions are Random and SlopeOne.
Many algorithms perform very well on average.
Furthermore,most algorithms perform very poorly in some cases: the maximum rank is at least 14 (out of 20) forall algorithms.
To illustrate the changes in algorithm performance across datasets, Figure 2 shows the normalizedmetric values for eight algorithms across 17 dataset splits.
Some algorithms tend to perform well(Item-KNN and SLIM-BPR) and others poorly (Random, TopPop), but no algorithm clearly dominatesfor all metrics and datasets.
This is a primary motivation for our meta-learning pipeline descirbedin Section 3: different algorithms perform well for different datasets on different metrics, so it isimportant to identify appropriate algorithms for each setting.
2.2 On the predictability of rec-sys algorithms
Can attributes of the rec-sys dataset be used to predict the performance of rec-sys algorithms?
Dataset meta-features.
We calculate 383 different meta-features to characterize each dataset.
These meta-features include statistics on the rating matrix—including basic statistics, the distributionbased features of Cunha et al.
[22], and landmark features [23]—which measure the performance ofsimple rec-sys algorithms on a subset of the training dataset.
Since these meta-features are used foralgorithm selection, they are calculated using only the training split of each dataset.
For more detailson the dataset meta-features, see Appendix A.4.
[Table]
Algorithm performance prediction.
Table 2 shows the meta-features that are most highlycorrelated with the performance (PREC@10) of each algorithm, using their default parameters.
Several meta-features aare highly-correlated with algorithm performance; one of the simplest metrics—the mean of the item rating count distribution—is highly correlated with performance of two rec-sysalgorithms.
This experiment motivates the design of RecZilla in the next section, which trains amodel using dataset meta-features to predict the performance of algorithms on new datasets.
As a toy-model version of RecZilla, we train three different meta-learner functions (XGBoost, KNN,and linear regression) using our meta-dataset, to predict performance metric PREC@10 for 10 rec-sysalgorithms with high average performance (see Appendix A for details).
We use leave-one-outevaluation for each meta-learner: one dataset family is held out for testing, while m are used fortraining.
Figure 3 shows the mean absolute error (MAE) of each meta-learner; these results areaggregated over 200 random samples of randomly-selected training dataset families.
MAE decreasesas more dataset families are added, suggesting that it is possible to estimate rec-sys algorithmperformance using dataset meta-features.
We also ﬁnd that performance metrics are not the only values that can be predicted with datasetmeta-features.
In particular, we ﬁnd that the runtime of rec-sys algorithms is also highly correlatedwith different meta-features.
Furthermore, we compute a simple measure of dataset hardness , whichwe compute as, given a performance metric, the maximum value achieved for that dataset across allalgorithms.
For example, if all 20 algorithms do not perform well on the MovieTweetings dataset,then we can expect that the MovieTweetings dataset is “hard”.
Once again, we ﬁnd that certaindataset meta-features are highly correlated with dataset hardness.
For more details on meta-featurecorrelation with algorithm runtimes and dataset hardness, see Appendix A.
The fact that dataset meta-features are correlated with algorithm performance, algorithm runtimes,and dataset hardness is a strong positive signal that meta-learning is worthwhile and useful in thecontext of recommender systems.
We explore this direction further in the next section.
RecZilla: Automated Algorithm Selection
In the previous section, we found that (1) the best algorithm and hyperparameters strongly dependon the dataset and user-chosen performance metric, and (2) the performance of algorithms can bepredicted from dataset meta-features.
Points (1) and (2) naturally motivate an algorithm selectionapproach to rec-sys powered by meta-learning.
In this section, we present RecZilla , which is motivated by a practical challenge: given a performancemetric and a new rec-sys dataset, quickly identify an algorithm and hyperparameters that performwell on this dataset.
This challenge arises in many settings—e.g., when selecting good baselinealgorithms for academic research, or when developing high-performing rec-sys algorithms for acommercial application.
We begin with an overview and then formally present our approach.
Overview.
As alluded to earlier, RecZilla is an algorithm selection approach powered by metalearning.
We use the results from the previous section as the meta-training dataset.
Given a userspeciﬁed performance metric, we train a meta-model that predicts the performance of each of aset of algorithms and hyperparameters on a dataset, by using meta-features of the dataset.
Given anew, unseen dataset, we compute the meta-features of the dataset, and then use the meta-model to predict the performance of each algorithm, returning the best algorithm according to the user-selectedperformance metric.
See Figure 1.
[Figure]
Preliminaries.
We start with notation for therec-sys problem.
Let D denote a rec-sys dataset,consisting of a set of user-item interactions, splitinto a training and validation set.
Let a denote arec-sys algorithm parameterized by a set of k ( a )hyperparameters !
2 H ( a ) ✓ R k
( a ) , which isalgorithm-dependent.
Suppose we train algorithm a on the training split of dataset D , using hyperparameters !
; we denote the performance of algorithm a with hyperparameters !
on dataset D as  ( a, ! , D ) 2 R .
The function ( · , · , · ) represents a performance metric for the 233 recommender system that is selected by the user;throughout this paper we refer to this functionand its numerical value simply as performance .
In this paper, larger values of  ( · , · , · ) indicates 237 better performance.
We report the normalizedperformance deﬁned as
[Equation]
minD , where P maxD (resp.
P minD ) are the max (resp.min) performance of any algorithm on dataset
D .
Next we deﬁne notation for the meta-learningproblem.
Given a ﬁxed performance function  ,let M = { ( D i , a i , !
i , y i ) } Mi =1 denote a meta- 245 dataset consisting of M tuples.
Each tuple consists of a dataset D , algorithm a parameterized by ! , and performance y , where y i ⌘  ( a i , !
i , D i ) .
Since many algorithms have a wide range of hyperparameters, we refer to the set of parameterizedalgorithms in the meta-dataset as S =
{ ( a i , !
i ) } Ni =1 .
We represent each dataset using vector d 2 R m , 249 where each element of d is a meta-feature of the dataset.
In this setting, the meta-learning task is toidentify a function that estimates the performance of parameterized algorithm ( a, ! ) on dataset D .
3.1 The RecZilla Algorithm Selection Pipeline
The RecZilla algorithm selection pipeline takes as input a meta-dataset M built with a user-chosenperformance function for the recommendation task  ( · , · , · ) .
RecZilla can accommodate any performance function that is a computable function of the performance metrics present in the meta-dataset.
The RecZilla pipeline proposed here returns both an algorithm a 2 A and a set of hyperparameters 256 !
2 H ( a ) , so that RecZilla can be used without additional hyperparameter optimization.
Since the meta-learning problem is in the low-data regime, to guard against overﬁtting, we select asubset of the algorithms that have good coverage over the training dataset.
We similarly select onlythe meta-features which are most predictive of the user-selected performance metric for the selectedalgorithms.
Below we outline each of the steps used to build the proposed RecZilla pipeline.
Algorithm subset selection: We select a subset S 0 ✓ S of n parameterized algorithms thatcollectively perform well on all datasets in the meta-training
dataset M , according to performancefunction  .
We aim to select an algorithm subset with high coverage over the set of knowndatasets, where coverage of a subset S 0 is deﬁned as
[Equation]
In other words, the coverage of subset S 0 is the normalized performance metric of the bestperforming algorithm in S 0 , averaged over all datasets D .
Selecting a subset with maximum
We compute %Diff as deﬁned in Section
2, as well as
Precision@
We report the mean and standard deviation across for
, for
[Table]
coverage is itself a difﬁcult problem; we use a greedy heuristic as follows.
We begin with S
= {}and iteratively add the parameterized algorithm ( a ⇤ , !
⇤ ) 2 arg max ( a, ! )
2 S C ( S
[ { ( a, ! ) } ) to
S 0 until |S 0 | = n .
That is, we greedily ensure that the coverage is maximized at each step.
Meta-feature selection: Similar to the previous point, we select a subset of meta-features with
good coverage over the meta-training dataset M ,
where here coverage
is
deﬁned
in
terms of thecorrelation
between
algorithm performance
and each meta-feature (see Appendix A.4 for details
Let M : D !
R m denote the resulting function that maps a dataset to a vector of meta-features.
Meta-learning: We learn a function f : R m !
R n , where f ( d ) =
ˆ y is a vector of the estimated
performance metric of all parameterized algorithms in S on dataset meta-features d .
Note that the RecZilla pipeline has two hyperparameters: n , the number of parameterized algorithmsin S 0 ; and m , the number of dataset meta-features used by the meta-learner.
In our experiments, werun an ablation study with both n and m , as well as different functions f for the meta-learning model.
Using RecZilla for Algorithm Selection.
After developing the RecZilla pipeline, we use thefollowing steps to select an algorithm for a new dataset D 0 :
Calculate m meta-features of the dataset dM ( D 0 ) .2.
Estimate the performance of all parameterized algorithms: yf ( d 0 ) .3.
Return the parameterized algorithm in S 0 with the best estimated performance.3.2 Experiments
In this section, we evaluate the end-to-end RecZilla pipeline.
We start by describing the speciﬁcversions of RecZilla used in our experiments.
We use four different meta-learning functions within
RecZilla: XGBoost [12], linear regression, k -nearest neighbors, and uniform random.
For KNN, weset k = 5 and use the L 2 distance from the selected meta-features.
Experimental setup.
Focusing only on performance metric to PREC@10, we build a meta-dataset
M using all rec-sys datasets, algorithms, and meta-features described in Section 2.
We ﬁrst use thealgorithm selection and meta-feature selection procedures described above to select n = m =
100parameterized algorithms and meta-features for PREC@10.
For all experiments we use the best10 parameterized algorithms selected during this process.
We vary both the number of trainingmeta-datapoints and meta-features used by RecZilla; the datapoints and features are randomlyselected over 50 random trials.
All RecZilla meta-learners are evaluated using leave-one-dataset-outevaluation: we iteratively select each dataset family as the meta-test dataset, and run the full RecZillapipeline using the remaining datasets as the meta-training data.
Splitting on dataset families ratherthan datasets ensures that there is no test data leakage.
Then for each dataset D in the test set,we compare the performance metric of the predicted best parameterized algorithm ( a 0 , ! 0 ) to theperformance metric of the ground-truth best algorithm y
⇤ , using the percentage-difference-from-best:% Diff = 100 ⇥ ( y ⇤   ( D, a 0 , ! 0 )) /y
⇤ .
% Diff is between 0 and 100, and smaller values indicatebetter performance.
Results and discussion.
In Table 3, we compare RecZilla with two prior algorithm selectionapproaches: cunha2018 [26] and cf4c4-meta [24], which are a comprehensive depiction of allprior work (see Appendix B.4 for justiﬁcation, and for details of the experiment).
Furthermore, note that cunha2018 has no open-source code, and cf4c4-meta only has code in R. We ﬁnd that
RecZilla outperforms the other two approaches in both %Diff and in terms of the PREC@10 value ofthe rec-sys algorithm outputted by each meta-learning algorithm.
In Appendix C.3, Figure 5 (left) shows % Diff vs. the size of the meta-training set, and Figure5 (right) shows the results of an ablation study on the number of selected meta-features m .
See
Appendix C.3 for more details and discussion.
Pre-trained RecZilla models.
We release pre-trained RecZilla models for PREC@10, NDCG@10,and Item-hit Coverage@10, trained with XGBoost on all 18 datasets, with algorithms n = 10 andmeta-features m = 10 .
We also include a RecZilla model that predicts the Pareto-front of PREC@10and model training time, so that users can select their desired trade-off between performance andruntime.
Finally, we include a pipeline so that users can choose a metric from the list of 315 (or anydesired combination of the 315 metrics) and train the resulting RecZilla model.
Conclusions, Limitations, and Broader Impact
In this work, we conducted the ﬁrst large-scale study of rec-sys approaches: we compared 24 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics.
We showed that for a givenperformance metric, the best algorithm and hyperparameters highly depend on the dataset.
We alsoﬁnd that various meta-features of the datasets are predictive of algorithmic performance and runtimes.
Motivated by these ﬁndings, we created RecZilla, the ﬁrst metric-independent, hyperparameter-awarealgorithm selection approach to recommender systems.
Through empirical evaluation, we show thatgiven a user-deﬁned metric, RecZilla effectively predicts high-performing algorithms and hyperparameters for new, unseen datasets, substantially reducing the need for human involvement.
We notonly release our code and pretrained RecZilla models, but we also release the raw experimental resultsso that users can train new RecZilla models on their own test metrics of interest.
This codebase,which includes a uniﬁed API for 85 datasets and 24 algorithms, may be of independent interest.
Limitations.
While our work progresses prior work along several axes, there are still avenues forimprovement.
First, the meta-learning problem in RecZilla is low-data.
Although we added nearly allcommon rec-sys research datasets into RecZilla, the result is still only 85 meta-datapoints (datasets).
While we guarded against over-ﬁtting to the training data in numerous ways, RecZilla can still beimproved by more training data.
Therefore, as new recommender system datasets are released inthe future, our hope is to add them to our API, so that RecZilla continuously improves over time.
Similarly, our hope is to add the most recent high-performing rec-sys approaches to our work, aswell as algorithms released in the future.
This includes adding neural network-based approaches,in addition to the six that we have already included.
Another limitation is that RecZilla does notdirectly predict the performance of hyperparameters for algorithms on a given dataset.
Although caremust be taken to not overﬁt, modifying RecZilla to predict the performance of an algorithm togetherwith a set of hyperparameters is an interesting avenue for future work.
Finally, the magnitude of ourevaluation ( 84 850 models trained) leaves our meta-data susceptible to biases based on experimentsuccess/failures.
While we ﬁxed many common errors such as out-of-memory errors, it was infeasibleto give each experiment speciﬁc attention.
Therefore, RecZilla may have higher uncertainty for thedatasets and algorithms that are more likely to fail.
An interesting future improvement to RecZillawould be to predict the likelihood that a new dataset will successfully train on a new dataset.
Broader impact.
Our work is “meta-research”: there is not one speciﬁc application that wetarget, but our work makes it substantially easier for researchers and practitioners to quickly trainrecommender system models when given a new dataset.
On the research side, this is a net positivebecause researchers can much more easily include baselines, comparisons, and run experiments onlarge numbers of datasets, all of which lead to more principled empirical comparisons.
On the appliedside, our day-to-day lives are becoming more and more inﬂuenced by recommendations generatedfrom machine learning models, which comes with pros and cons.
These recommendations connectusers with needed items that they would have had to spend time searching for [52].
Although theserecommendations may lead to harmful effects such as echo chambers [36, 53], techniques to identifyand mitigate harms are improving [40, 67].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
[The main claims in the abstract and introduction reﬂectthe paper’s contributions and scope.](b) Did you describe the limitations of your work?
[Yes] [See Section 4.](c)
Did you discuss any potential negative societal impacts of your work?
[Yes] [SeeSection 4.](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] [We read the ethics review guidelines and ensured our paper conforms to
them.]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
[We did notinclude theoretical results.]
(b) Did you include complete proofs of all theoretical results?
[N/A]
[We did not includetheoretical results.]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
[Weinclude the code, data, and instructions to reproduce the results here: https://anonymous.4open.science/r/anon-reczilla-51FC .]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] [See Section 2 and Appendix A.]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] [See Section 3.]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] [We include this information in
Section 2].4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] [See Section 2 and
Appendix A.]
(b) Did you mention the license of the assets?
[N/A] Our experiments were conductedonly on publicly available datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We did not include new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
Our experiments were conducted only on publicly available
[Table]
datasets.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
Our experiments were conducted only onpublicly available datasets.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[We did not conduct research with human subjects.]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[We did not conduct research with human
[Table]
subjects.]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[We did not conduct research with human
[Table]
subjects.]
[Table]