On the Generalizability and Predictability of Recommender Systems
Anonymous Author(s)
AfﬁliationAddress email
Abstract
While other areas of machine learning have seen more and more automation,designing a high-performing recommender system still requires a high level ofhuman effort.
Furthermore, recent work has shown that modern recommendersystem algorithms do not always improve over well-tuned baselines.
A naturalfollow-up question is, “how do we choose the right algorithm for a new datasetand performance metric?”
In this work, we start by giving the ﬁrst large-scalestudy of recommender system approaches by comparing 18 algorithms and 100sets of hyperparameters across 85 datasets and 315 metrics.
We ﬁnd that thebest algorithms and hyperparameters are highly dependent on the dataset andperformance metric, however, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets.
Motivatedby these ﬁndings, we create RecZilla, a meta-learning approach to recommendersystems that uses a model to predict the best algorithm and hyperparameters fornew, unseen datasets.
By using far more meta-training data than prior work,
RecZilla is able to substantially reduce the level of human involvement whenfaced with a new recommender system application.
We not only release our codeand pretrained RecZilla models, but also all of our raw experimental results, sothat practitioners can train a RecZilla model for their desired performance metric:https://anonymous.4open.science/r/anon-reczilla-51FC .
Introduction
Due to the large computational resources for training machine learning models, researchers havefound many ways to repurpose existing computation.
For example, it is common to start with apretrained ImageNet classiﬁcation model for computer vision tasks [30, 72, 48], or a pretrained BERTmodel for natural language tasks [31, 63].
These approaches work well because the core problems arelargely homogeneous; for example, any computer vision model at its core must be able to distinguishedges, colors and shapes.
Even a task-speciﬁc architecture can be found automatically through neuralarchitecture search [35], since the building blocks such as convolutional layers stay the same.
On the other hand, recommender system (rec-sys) research has followed a different trajectory: despitetheir widespread usage across many e-commerce, social media, and entertainment companies such as
Amazon, YouTube, and Netﬁx [19, 39, 74], there is far less work in reusing models or automating theprocess of selecting models.
Many rec-sys techniques are designed and optimized with just a singledataset in mind [39, 46, 19, 56, 78].
Intuitively, this might be because each rec-sys application ishighly unique based on the dataset and the target metric.
For example, a typical user session looksvery different among e.g. YouTube, Home Depot, and AirBnB [19, 56, 46].
However, this intuitionhas not been formally established.
Furthermore, recent work has shown that neural recommender system algorithms do not always improve over well-tuned baselines such as k -nearest neighbor and
[Table]
RecZilla pipeline is built using a meta-dataset M that includes many different performance metrics evaluated on many different rec-sys algorihtms on many different datasets; we estimate algorithm performance using dataset meta-features.
To avoid over-ﬁtting and reduce runtime, users can limit the number of algorithms and meta-features to constants n and m .
matrix factorization [26].
A natural question is then, “how do we choose the right algorithm for anew dataset and performance metric?”
In this work, we show that the best algorithm and hyperparameters are highly dependent on thedataset and user-deﬁned performance metric.
Speciﬁcally, we run the ﬁrst large-scale study ofrec-sys approaches by comparing 18 algorithms across 85 datasets and 315 metrics.
For each datasetand algorithm pair, we test up to 100 hyperparameters (given a 10 hour time limit per pair).
Thecodebase that we release, which includes a uniﬁed API for a large, diverse set of algorithms, datasets,and metrics, may be of independent interest.
We show that the algorithms do not generalize – theset of algorithms which perform well changes substantially across dataset and across performancemetrics.
Furthermore, the best hyperparameters of a rec-sys algorithm on one dataset often performsigniﬁcantly worse than the best hyperparameters on a different dataset.
Although we show that thereare no universal algorithms that work well on most datasets, we do show that various meta-featuresof the dataset can be used to predict the performance of rec-sys algorithms.
In fact, the same metafeatures are also predictive of the runtime of rec-sys algorithms as well as the “dataset hardness” –how challenging it is to ﬁnd a high-performing model on a particular dataset.
Motivated by these ﬁndings, we introduce RecZilla, a meta-learning-based algorithm selectionapproach (see Figure 1) inspired by SATzilla [80].
At the core of RecZilla is a model that, given auser-deﬁned performance metric, predicts the best rec-sys algorithm and hyperparameters for a newdataset based on meta dataset features such as number of users and items, and spectral properties ofthe interaction matrix.
We show that RecZilla quickly ﬁnds high-performing algorithms on datasets ithas never seen before.
While there has been prior work on meta-learning for recommender systems[24, 25], no prior work is metric-independent, searches for hyperparameters as well as algorithms,or considers more than nine dataset families.
By running an ablation study on the number of metatraining datasets, we show that more dataset families are crucial to the success of RecZilla.
We releaseready-to-use, pretrained RecZilla models for common test metrics, and we release the raw resultsfrom our large-scale study, along with code so that practitioners can easily train a new RecZilla modelfor their speciﬁc performance metric of interest.
Our contributions.
We summarize our main contributions below
• We run a large-scale study of recommender systems, showing that the best algorithm and hyperparameters are highly dependent on the dataset and user-deﬁned performance metric.
We also
show that dataset meta-features are predictive of the performance of algorithms.•
We create RecZilla, a meta-learning-based algorithm selection approach which, given a perfor-
mance metric, efﬁciently predicts the best algorithm and set of hyperparameters on new datasets.
• We release a public repository containing 85 datasets and 18 rec-sys algorithms, accessed througha uniﬁed API.
Furthermore, we release both pretrained RecZilla models, and raw data so thatusers can train a new RecZilla model on their desired metric
Related Work Recommender systems are a widely studied area of research [10].
Common approaches include k -nearest neighbors [1], matrix factorization [55, 61], and deep learning approaches[19, 39, 74].
For a survey on recommender systems, see [10, 4].
A recent meta-study showed that ofthe 12 published neural rec-sys approaches published at top conferences between 2015 and 2018, 11performed worse than well-tuned baselines (e.g. nearest neighbor search or linear models) [26].
Algorithm selection for recommender systems was ﬁrst studied in 2011 [50] by using a graphrepresentation of item ratings.
Follow-up work used dataset meta-features to select the best nearestneighbor and matrix factorization algorithms [34, 3, 41].
Subsequent work focused on improvingthe model and framework [25] including studying 74 meta-features systematically [21].
More recentapproaches from 2018 run meta-learning for recommender systems by casting the meta-problem itselfas a collaborative ﬁltering problem.
Performance is then estimated with subsampling landmarkers[22, 24, 23].
No prior work in algorithm selection for rec-sys includes open-source Python code.
There is also work on automated machine learning (AutoML) for recommender systems, withoutmeta-learning [79, 6, 44, 45].
Finally, we note that meta-learning across rec-sys datasets is not to beconfused with the body of work on meta-learning user preferences within a single rec-sys dataset[15, 16, 17].
To the best of our knowledge, no meta-learning or AutoML rec-sys paper has runexperiments on more than nine dataset families or four test metrics, and no prior work predictshyperparameters in addition to algorithms.
Analysis of Recommender Systems
In this section, we present a large-scale empirical study of rec-sys algorithms across a large, diverseset of datasets and metrics.
We assess the following two research questions.
Generalizability.
If a rec-sys algorithm or set of hyperparameters performs well on one datasetand metric, will it perform well on other datasets or on other metrics?
Predictability.
Given a metric, can various dataset meta-features be used to predict the perfor- mance of rec-sys algorithms?
Algorithms, datasets, and metrics implemented.
We run experiments with 18 rec-sys algorithms,including methods from recent literature and common baselines.
This includes ﬁve similarity andclustering-based methods: User-KNN [71], Item-KNN [73], P3Alpha [18], RP3Beta [69], and
Co-Clustering [37]; six Matrix-Factorization (MF) methods: MF-FunkSVD, MF-AsySVD [54],
MF-BPR [70], IALS [49], Pure-SVD, Non-negative matrix factorization (NMF) [20]; ﬁve methodsbased on linear models: Global-Effects, SLIM-BPR [7], SLIM-Elastic-Net [60], EASE-R [75], and
SlopeOne [59]; and two simple baselines: Random and Top-Pop.
These algorithms were chosen dueto their high performance, popularity, and speed.
For many algorithms, we used the implementationsfrom the codebase of Dacrema et al.
For full details of the algorithms, see Appendix A.
We run the algorithms on 85 datasets from 19 dataset “families”:
Amazon [68], Anime 1 , BookCrossing [84], CiaoDVD [57, 43], Dating (Libimseti.cz) [57, 58], Epinions [65, 64], FilmTrust [42],
Frappe [8], Gowalla [14], Jester2 [38], LastFM [11], MarketBias-Electronics and MarketBiasModCloth [77], MovieTweetings [33], Movielens [47], NetﬂixPrize [9], Here, a “dataset family”refers to an original dataset (such as Movielens), while “dataset” refers to a single train-test splitdrawn from the original dataset, which may be a small subset of the original.
We implemented themajority of rec-sys datasets commonly used for research, and to the best of our knowledge, this is thelargest number of rec-sys datasets accessible in a single open-source repository.
We use 23 different “base” metrics: ARHR, Average Popularity, Diversity Similarity, F1 Score, Gini
Index, Herﬁndahl Index, Hit Rate, Item Coverage, Item-hit Coverage, Precision (PREC), PrecisionRecall Min Denominator, Recall, MAP, MAP Min Denominator, Mean Inter-List Diversity, MRR,
NDCG, Novelty, Shannon Entropy, User Coverage, User-hit Coverage.
All of these metrics arecomputed at cutoffs { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 15 , 20 , 30 , 40 , 50 } , for a total of 315 different metrics. [Table]
See Appendix A.6 for more details.
These metrics include the most popular from the literature, andwe use the Dacrema et al.
[26] implementations.
We focus mainly on metrics related to rating- andranking-based metrics, such as NDCG and Precision.
Experimental design.
For each dataset, we compute a train and test split based on leave-last- kout (and our repository also includes splits based on global timestamp).
Given each of the abovealgorithms, we expose several hyperparameters and give ranges based on common values.
For eachdataset, we run each algorithm on a random sample of up to 100 hyperparameter sets.
Each algorithmis allocated a 10 hour limit for each dataset split; we train and test the algorithm with at most 100hyperparameter sets on [-ann1-highmem-
 Google Cloud instance,-]an n1-highmem-2 CPU, until the time limit is reached.
Each algorithm istrained on the train split, and the performance metrics are computed on the test split.
We refer to eachcombination of (algorithm, hyperparameter set, dataset) as an experiment .
By running 18 algorithms, up to 100 hyperparameters, and 85 datasets, this resulted in 84 769successful experiments, and by computing 315 metrics, our ﬁnal meta-dataset of results includesmore than 26 million evaluations.
We give a more detailed look at the breakdown of experiments in
Appendix A, and we discuss any potential biases in the resulting dataset in Section 4.
2.1 On the generalizability of rec-sys algorithms
If a rec-sys algorithm or set of hyperparameters performs well on one dataset and metric, will itperform well on other datasets or on other metrics?
Our ﬁrst observation is that all algorithms perform well on some datasets, and poorly on others.
Firstwe identify the best-performing hyperparameter set for each (algorithm, dataset) pair—to simulatehyperparameter optimization using our meta-dataset.
We then rank all algorithms for each dataset,according to several performance metrics.
If we focus on a single metric, then most algorithms are ranked ﬁrst according to this metric on atleast one dataset.
For example, for metric PREC@10, 14 out or 18 algorithms are ranked ﬁrst on atleast one dataset; the exceptions are CoClustering, SlopeOne, GlobalEffects, and Random; the sameis true for many other metrics and cutoffs (see Table 5 in Appendix C).
Average performance is more varied: some algorithms tend to perform better than others.
Table 1shows the mean, min (best) and max (worst) ranking of all 18 algorithms over all dataset and allaccuracy and hit-rate metrics.
This includes metrics NDCG, precision, recall, Prec.-Rec.-Min-density,hit-rate, F1, MAP, MAP-Min-density, ARHR, and MRR (see Appendix A.6 for descriptions of thesemetrics).
Nearly all algorithms are ranked ﬁrst for at least one metric, on at least one dataset; theonly exception is Random, which has a minimum rank 2.
Many algorithms perform very well onaverage; interestingly, the three algorithms with the highest average ranking each come from differentalgorithm families: Item-KNN is a similarity-based metric, SLIM-BPR is based on linear models,and SVD is a matrix factorization method.
Furthermore, most algorithms perform very poorly insome cases: the maximum rank is at least 15 (out of 18) for all algorithms.
To illustrate the changes in algorithm performance across datasets, Figure 2 shows the normalizedmetric values for eight algorithms across 17 dataset splits.
Some algorithms tend to perform well(Item-KNN and SLIM-BPR) and others poorly (Random, TopPop), but no algorithm clearly dominatesfor all metrics and datasets.
This is a primary motivation for our meta-learning pipeline descirbedin Section 3: different algorithms perform well for different datasets on different metrics, so it isimportant to identify appropriate algorithms for each setting.
2.2 On the predictability of rec-sys algorithms
Can attributes of the rec-sys dataset be used to predict the performance of rec-sys algorithms?
Dataset meta-features.
We calculate 383 different meta-features to characterize each dataset.
These meta-features include statistics on the rating matrix—including basic statistics, the distributionbased features of Cunha et al.
[21], and landmark features [22]—which measure the performance ofsimple rec-sys algorithms on a subset of the training dataset.
Since these meta-features are used foralgorithm selection, they are calculated using only the training split of each dataset.
For more detailson the dataset meta-features, see Appendix A.4.
Algorithm performance prediction.
Table 2 shows the meta-features that are most highlycorrelated with the performance (PREC@10) of each algorithm, using their default parameters.
Several meta-features aare highly-correlated with algorithm performance; one of the simplest metrics—the mean of the item rating count distribution—is highly correlated with performance of two rec-sysalgorithms.
This experiment motivates the design of RecZilla in the next section, which trains amodel using dataset meta-features to predict the performance of algorithms [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] As a toy-model version of RecZilla we trainthree different meta-learner functions (XGBoost, KNN and linear regression) using ourmeta-dataset, to predict performance metric
PREC@10 for 10 rec-sys algorithms with highaverage performance (see Appendix A for details).
We use leave-one-out leave-one-out evaluation for each meta-learner: one dataset family is held out for testing, while m are usedfor training.
Figure 3 shows the mean absolute error (MAE) of each meta-learner; theseresults are aggregated over 200 random samples of randomly-selected training dataset families.
MAE decreases as more dataset families are added, suggesting that it is possible toestimate rec-sys algorithm performance usingdataset meta-features.
We also ﬁnd that performance metrics are not theonly values that can be predicted with datasetmeta-features.
In particular, we ﬁnd that theruntime of rec-sys algorithms is also highly correlated with different meta-features.
Furthermore, we compute a simple measure of datasethardness , which we compute as, given a performance metric, the maximum value achievedfor that dataset across all algorithms.
For example, if all 18 algorithms do not perform well onthe MovieTweetings dataset, then we can expectthat the MovieTweetings dataset is “hard”.
Once again, we ﬁnd that certain dataset meta-features arehighly correlated with dataset hardness.
For more details on meta-feature correlation with algorithmruntimes and dataset hardness, see Appendix A.
The fact that dataset meta-features are correlated with algorithm performance, algorithm runtimes,and dataset hardness is a strong positive signal that meta-learning is worthwhile and useful in thecontext of recommender systems.
We explore this direction further in the next section.
RecZilla: Automated Algorithm Selection
In the previous section, we found that (1) the best algorithm and hyperparameters strongly dependon the dataset and user-chosen performance metric, and (2) the performance of algorithms can bepredicted from dataset meta-features.
Points (1) and (2) naturally motivate an algorithm selectionapproach to rec-sys powered by meta-learning.
In this section, we present RecZilla , which is motivated by a practical challenge: given a performancemetric and a new rec-sys dataset, quickly identify an algorithm and hyperparameters that performwell on this dataset.
This challenge arises in many settings—e.g., when selecting good baseline algorithms for academic research, or when developing high-performing rec-sys algorithms for acommercial application.
We begin with an overview and then formally present our approach.
Overview.
As alluded to earlier, RecZilla is an algorithm selection approach powered by metalearning.
We use the results from the previous section as the meta-training dataset.
Given a userspeciﬁed performance metric, we train a meta-model that predicts the performance of each of aset of algorithms and hyperparameters on a dataset, by using meta-features of the dataset.
Given anew, unseen dataset, we compute the meta-features of the dataset, and then use the meta-model topredict the performance of each algorithm, returning the best algorithm according to the user-selectedperformance metric.
See Figure 1.
Preliminaries.
We start with notation for the rec-sys problem.
Let D denote a rec-sys dataset,consisting of a set of user-item interactions, split into a training and validation set.
Let a denotea rec-sys algorithm parameterized by a set of k ( a ) hyperparameters !
2 H ( a ) ✓ R k
( a ) , whichis algorithm-dependent.
Suppose we train algorithm a on the training split of dataset D , usinghyperparameters !
; we denote the performance of algorithm a with hyperparameters !
on dataset D as  ( a, ! , D ) 2 R .
The function  ( · , · , · ) represents a performance metric for the recommendersystem that is selected by the user; throughout this paper we refer to this function and its numericalvalue simply as performance .
In this paper, larger values of  ( · , · , · ) indicates better performance.
We report the normalized performance deﬁned as
[Equation]
where P maxD (resp.
P minD ) are the max (resp. min) performance of any algorithm on dataset D .
Next we deﬁne notation for the meta-learning problem.
Given a ﬁxed performance function  ,let M = { ( D i , a i , !
i , y i ) } Mi =1 denote a meta-dataset consisting of M tuples.
Each tuple consistsof a dataset D , algorithm a parameterized by ! , and performance y , where y i ⌘  ( a i , !
i , D i ) .
Since many algorithms have a wide range of hyperparameters, we refer to the set of parameterizedalgorithms in the meta-dataset as S =
{ ( a i , !
i ) } Ni =1 .
We represent each dataset using vector d 2 R m , 241 where each element of d is a meta-feature of the dataset.
In this setting, the meta-learning task is toidentify a function that estimates the performance of parameterized algorithm ( a, ! ) on dataset D .
3.1 The RecZilla Algorithm Selection Pipeline
The RecZilla algorithm selection pipeline takes as input a meta-dataset M built with a user-chosenperformance function for the recommendation task  ( · , · , · ) .
RecZilla can accommodate any performance function that is a computable function of the performance metrics present in the meta-dataset.
The RecZilla pipeline proposed here returns both an algorithm a 2 A and a set of hyperparameters 248 !
2 H ( a ) , so that RecZilla can be used without additional hyperparameter optimization.
Since the meta-learning problem is in the low-data regime, to guard against overﬁtting, we select asubset of the algorithms that have good coverage over the training dataset.
We similarly select onlythe meta-features which are most predictive of the user-selected performance metric for the selectedalgorithms.
Below we outline each of the steps used to build the proposed RecZilla pipeline.
Algorithm subset selection: We select a subset S 0 ✓ S of n parameterized algorithms that collectively perform well on all datasets in the meta-training dataset M , according to performance
function  .
We aim to select an algorithm subset with high coverage over the set of known datasets, where coverage of a subset S 0 is deﬁned as
[Equation]
In other words, the coverage of subset S 0 is the normalized performance metric of the bestperforming algorithm in S 0 , averaged over all datasets D .
Selecting a subset with maximumcoverage is itself a difﬁcult problem; we use a greedy heuristic as follows.
We begin with S
= {}and iteratively add the parameterized algorithm ( a ⇤ , !
⇤ ) 2 arg max ( a, ! )
2 S C ( S
[ { ( a, ! ) } ) to
S 0 until |S 0 | = n .
That is, we greedily ensure that the coverage is maximized at each step.
[Figure]
Meta-feature selection: Similar to the previous point, we select a subset of meta-features withgood coverage over the meta-training dataset M , where here coverage is deﬁned in terms of thecorrelation between algorithm performance and each meta-feature (see Appendix A.4 for details).
Let M : D !
R m denote the resulting function that maps a dataset to a vector of meta-features.
Meta-learning: We learn a function f : R m !
R n , where f ( d ) =
ˆ y is a vector of the estimatedperformance metric of all parameterized algorithms in S on dataset meta features d .
Note that the RecZilla pipeline has two hyperparameters: n , the number of parameterized algorithmsin S 0 ; and m , the number of dataset meta-features used by the meta-learner.
In our experiments, werun an ablation study with both n and m , as well as different functions f for the meta-learning model.
Using RecZilla for Algorithm Selection.
After developing the RecZilla pipeline, we use thefollowing steps to select an algorithm for a new dataset D 0 :
Calculate m meta-features of the dataset dM ( D 0 ) .2.
Estimate the performance of all parameterized algorithms: yf ( d 0 ) .
Return the parameterized algorithm in S 0 with the best estimated performance. 3.2 Experiments
In this section, we evaluate the end-to-end RecZilla pipeline.
We start by describing the speciﬁcversions of RecZilla used in our experiments.
We use four different meta-learning functions within
RecZilla: XGBoost [12], linear regression, k -nearest neighbors, and uniform random.
For KNN, weset k = 5 and use the L 2 distance from the selected meta-features.
Experimental setup.
Focusing only on performance metric to PREC@10, we build a meta-dataset
M using all rec-sys datasets, algorithms, and meta-features described in Section 2.
We ﬁrst use thealgorithm selection and meta-feature selection procedures described above to select n = m =
100parameterized algorithms and meta-features for PREC@10.
For all experiments we use the best10 parameterized algorithms selected during this process.
We vary both the number of trainingmeta-datapoints and meta-features used by RecZilla; the datapoints and features are randomlyselected over 50 random trials.
All RecZilla meta-learners are evaluated using leave-one-dataset-outevaluation: we iteratively select each dataset family as the meta-test dataset, and run the full RecZillapipeline using the remaining datasets as the meta-training data.
Splitting on dataset families ratherthan datasets ensures that there is no test data leakage.
Then for each dataset D in the test set,we compare the performance metric of the predicted best parameterized algorithm ( a 0 , ! 0 ) to theperformance metric of the ground-truth best algorithm y
⇤ , using the percentage-difference-from-best:% Diff = 100 ⇥ ( y ⇤   ( D, a 0 , ! 0 )) /y
⇤ .
% Diff is between 0 and 100, and smaller values indicatebetter performance.
Results and discussion.
Figure 4 (Left) shows % Diff vs. the size of the meta-training set, and Figure 4 (Right) shows the results of an ablation study on the number of selected meta-features m ;all results are aggregated over all leave-one-out folds and 50 random trials.
Generally, XGBoost and
KNN outperform the linear model and the random baseline, with XGBoost achieving top performancewhen using 10 meta-features and the maximum number of training datasets. Furthermore, the numberof datasets in the meta-training set matters more than the meta-learning model itself.
For example,the improvement of XGBoost from 4 to 10 and to 18 training datasets is larger than the difference inperformance between XGBoost and KNN at 4 and 10 training datasets, respectively.
Finally, we ﬁndthat the optimal number of meta-features for XGBoost and KNN peaks between 10 and 40.
Pre-trained RecZilla models.
We release pre-trained RecZilla models for PREC@10, NDCG@10,and Item-hit Coverage@10, trained with XGBoost on all 18 datasets, with algorithms n = 10 andmeta-features m = 10 .
We also include a RecZilla model that predicts the Pareto-front of PREC@10and model training time, so that users can select their desired trade-off between performance andruntime.
Finally, we include a pipeline so that users can choose a metric from the list of 315 (or anydesired combination of the 315 metrics) and train the resulting RecZilla model.
Conclusions, Limitations, and Broader Impact
In this work, we conducted the ﬁrst large-scale study of rec-sys approaches: we compared 18 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics.
We showed that for a givenperformance metric, the best algorithm and hyperparameters highly depend on the dataset.
We alsoﬁnd that various meta-features of the datasets are predictive of algorithmic performance and runtimes.
Motivated by these ﬁndings, we created RecZilla, the ﬁrst metric-independent, hyperparameter-awarealgorithm selection approach to recommender systems.
Through empirical evaluation, we show thatgiven a user-deﬁned metric, RecZilla effectively predicts high-performing algorithms and hyperparameters for new, unseen datasets, substantially reducing the need for human involvement.
We notonly release our code and pretrained RecZilla models, but we also release the raw experimental resultsso that users can train new RecZilla models on their own test metrics of interest.
This codebase,which includes a uniﬁed API for 85 datasets and 18 algorithms, may be of independent interest.
Limitations.
While our work progresses prior work along several axes, there are still avenues forimprovement.
First, the meta-learning problem in RecZilla is low-data.
Although we added nearly allcommon rec-sys research datasets into RecZilla, the result is still only 85 meta-datapoints (datasets).
While we guarded against over-ﬁtting to the training data in numerous ways, RecZilla can still beimproved by more training data.
Therefore, as new recommender system datasets are released in thefuture, our hope is to add them to our API, so that RecZilla continuously improves over time.
Anotherpossible solution to increase the meta-training dataset is to create subsets of existing rec-sys datasets.
Another limitation is that RecZilla does not directly predict the performance of hyperparametersfor algorithms on a given dataset.
Although care must be taken to not overﬁt, modifying RecZillato predict the performance of an algorithm together with a set of hyperparameters is an interestingavenue for future work.
Finally, the magnitude of our evaluation ( 78 929 rec-sys models trained)leaves our meta-data susceptible to biases based on experiment success/failures.
While we ﬁxedmany common errors such as out-of-memory errors, it was infeasible to give each experiment speciﬁcattention.
Therefore, RecZilla may have higher uncertainty for the datasets and algorithms that aremore likely to fail.
An interesting future improvement to RecZilla would be to predict the likelihoodthat a new dataset will successfully train on a new dataset.
Broader impact.
Our work is “meta-research”: there is not one speciﬁc application that wetarget, but our work makes it substantially easier for researchers and practitioners to quickly trainrecommender system models when given a new dataset.
On the research side, this is a net positivebecause researchers can much more easily include baselines, comparisons, and run experiments onlarge numbers of datasets, all of which lead to more principled empirical comparisons.
On the appliedside, our day-to-day lives are becoming more and more inﬂuenced by recommendations generatedfrom machine learning models, which comes with pros and cons.
These recommendations connectusers with needed items that they would have had to spend time searching for [52].
Although theserecommendations may lead to harmful effects such as echo chambers [36, 53], techniques to identifyand mitigate harms are improving [40, 66].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
[The main claims in the abstract and introduction reﬂect the paper’s contributions and scope.] (b) Did you describe the limitations of your work?
[Yes] [See Section 4.]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] [See Section 4.] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] [We read the ethics review guidelines and ensured our paper conforms to
them.]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
[We did not include theoretical results.]
(b) Did you include complete proofs of all theoretical results?
[N/A]
[We did not include theoretical results.]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes]
[We include the code, data, and instructions to reproduce the results here: https://anonymous.4open.science/r/anon-reczilla-51FC .]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] [See Section 2 and Appendix A.]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] [See Section 3.]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] [We include this information inSection 2].
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] [See Section 2 andAppendix A.]
(b) Did you mention the license of the assets?
[N/A] Our experiments were conductedonly on publicly available datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]We did not include new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
Our experiments were conducted only on publicly availabledatasets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
Our experiments were conducted only onpublicly available datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[We did not conduct research with human subjects.]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[We did not conduct research with human
subjects.]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[We did not conduct research with human
[Table]
subjects.
[Table]