Paper: Basis Function Encoding of Numerical Features in Factorization Machines for Improved Accuracy (ICLR-2024)

Reviewer comment (Evaluation):
The authors mentioned the computational efficiency of the proposed approach. Is there a comparison about the computing time?

Author response:
Regarding computational efficiency, we indeed have not conducted such studies. But it heavily depends on having a very efficient implementation that integrates well with the rest of the code with little overhead. 

If it's interesting for you, in the product where we conducted the A/B tests reported in the paper there was no visible effect on ranking latency, and ~2% increase in the time it takes to execute the training pipeline. But this is not very informative, since evaluating the cubic B-Spline basis and adding up the four vectors in the corresponding field is just one of the many things the pipeline does.

Reviewer comment (Experiments):
What are the functions $p_i$ in the synthetic experiment? They appear to be very similar in shape to cubic splines; are they themselves cubic splines? If so, is the observed error of the proposed method simply due to a finite training set?

Author response:
The functions $p_i$ are of the form $\frac{a}{(a + b + x)^{1.5}} + \frac{a}{(a + b)^{1.5}} \frac{5}{\cosh(x - \beta)}$, where $a, b, \beta$ are parameters. We wanted a function with a "bump" somewhere that we can move and change its height using the parameters. Since we put the code to reproduce all experimental results, interested readers may download the supplemental material and see exactly what these functions are. If you believe it’s essential in the paper itself, we’ll add it to the figure’s description (you can find it in the ctr_curve_learning notebook, cells 5-7)

Reviewer comment (Theory):
Based on equation (1), it seems that an FM is a feature embedding plus a second order polynomial regression without "diagonal" (i.e., $x_i^2$) second order terms. There may be other restrictions on the coefficients for the resulting quadratic based on the form of equation (1). While the introduction explains why these methods are preferred over larger models like neural networks for fast inference, the reviewer asks whether there is a specific benefit or motivation for this particular form of model that makes it preferable to a general quadratic function. Either a discussion of this or a pointer to a specific reference would make the paper more accessible to readers unfamiliar with FMs.

Author response:
See our general comment, part (3). It’s “almost” a general quadratic function, but in factorized form. The reason the diagonal is missing is simple - the effect of single features is already captured by the linear part, so the quadratic part’s purpose is modeling the effect of pairs of features. We hope that what we added in the appendix which we referred to from Section 2.1 indeed makes the paper more accessible. Please see the 2010 paper 'Factorization Machines' by Stephen Rendle for more details.

Reviewer comment (Evaluation):
In the online system, CTR prediction error measures the accuracy of model prediction, while the ranking performance between advertisements needs to be measured by the AUC metric.

Author response:
It is important to understand that this is an online A/B test, rather than the evaluation on a data-set. In this case, metrics such as LogLoss or AUC are not informative at all. This is mainly because in advertising ads are **not** selected based on CTR, but based on the expected revenue $\mathrm{CTR} \cdot \mathrm{bid}$. The better we predict CTR, the better we predict the expected revenue, and the better our marketplace performs.  One model may cause more ads from a high-CTR category to win the auction, whereas its competitor may cause more ads with a lower CTR but higher bid ads to win auctions. Since the set of ads displayed to the user may be radically different between the two experimental environments, the "test sets" are different, metrics such as LogLoss or AUC are not comparable on real traffic.

CTR prediction error, on the other hand, is much more informative. When the model over-predicts, it causes ads to unjustifiably win auctions, and due to the over-prediction, the true CTR will be lower than the predicted one. When the model under-predicts, it causes some ads to unjustifiably lose auctions, they will not be displayed to the user, and we will not observe these under-predictions at all. Thus, on average, models over-predict on real traffic that is displayed to the user, and the amount of over-prediction is an informative indicator of the model's accuracy.

Of course, the best indicator is revenue. However, measuring revenue requires an experimentation environment where campaign budgets are not shared between the two experimental environments, otherwise the environment with the significantly less accurate model "steals" budgets from the environment with the significantly more accurate model. As is apparent in the paper, the prediction error before changing the field's representation was quite high. Such an environment has limited availability, and our team uses it only for very few high priority experiments. Thus, we were not able to report the revenue increase. Please see  [this](https://arxiv.org/abs/2012.08724) and [this](https://arxiv.org/abs/1605.09171) papers about the budget isolation issue.

The above are not obvious for readers not experienced in online advertising, but we were not able to find a good reference explaining all of the above concerns in one place. We hope that this explanation helps the reviewer.

Reviewer comment (Experiments):
How to choose the number of break-points for a specific FM, and it requires sensitivity analysis. In addition, the author states that the choice of break-points may vary between fields, so the complexity of the choice seems to reduce the practicality of the proposed method.

Author response:
We kindly ask the reviewer to read the general comment above. It answers question (1). Regarding question (2), it can indeed be complex if we want to win a Kaggle competition - take a dataset and apply it to all numerical fields by customizing the appropriate number of break points to every field. However:
(1) we can see in the numerical experiments on public datasets, that even setting the number of bins to the same value for all numerical fields already provides a significant boost over binning.
(2) in real recommender systems, fields are often added or changed incrementally, rather than all at once. So a data scientist working on feature engineering for a particular field can conduct the required experiments to choose the appropriate number of break points for their field.

Reviewer comment (Experiments):
An important question that I could not find the answer in the paper is that why do we need to transform features into step function like features? Instead you can use neural networks as an example to produce a representation for each given feature vector and then the learned representation can be employed to train a model. This probably can be added as a baseline to the evaluation section.

Author response:
Regarding the second weakness of using step functions. This is not our contribution, but a standard practice with FMs, since they can only represent a linear function of any given feature. Binning is a standard non-parametric technique that allows simple models  to approximately learn arbitrary function using a piecewise-constant representation. We also refer the reader to the paper "Practical Lessons from Predicting Clicks on Ads at Facebook" where additional, more advanced binning techniques are discussed in Section 3.1. Our contribution is an improvement over binning that is only slighly more complex, but significanlty more accurate.

Reviewer comment (Experiments):
The utilization of hyperparameter search for small datasets by the authors may not be practical for real-world use cases with large datasets. A preferable approach would involve conducting ablation studies, especially on factors like step-size, to guide the selection of values for practical use cases.

Author response:
We kindly ask the reviewer to read the general comment above. It addresses some of the concerns pointed out in the weaknesses section.

One point regarding hyperparameter search. In practice, hyperparameter search can be, and **is** done in practice. For example, please look at the following works:
1. Michal Aharon, Amit Kagian, and Oren Somekh. 2017. Adaptive Online Hyper-Parameters Tuning for Ad Event-Prediction Models. In Proceedings of the 26th International Conference on World Wide Web Companion (WWW '17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 672–679. https://doi.org/10.1145/3041021.3054184
2. Parker-Holder, J., Nguyen, V. and Roberts, S.J., 2020. Provably efficient online hyperparameter optimization with population-based bandits. Advances in neural information processing systems, 33, pp.17200-17211.

Moreover, one hyperparameter configuration can be better for bins, whereas another can be better for splines. Especially parameters like the step-size, that heavily depend on the variance of gradients, that varies greatly when changing the representation basis of choice. Thus, with one set of hyperparameters we could conclude that bins outperform splines, whereas with another one, splines outperform bins. We wanted to demonstrate that the best splines can do is better than the best that bins can do.

Reviewer comment (Novelty):
Novelty of spline: spline-based regressions/approximations also seems to be have been around for very long, and we should already have abundant techniques to understand its statistical/generalization properties (e.g., how choice of bins may impact training and test performance tradeoff). I fail to see any new contribution from this paper.

Author response:
Please refer to the general comment above, since it addresses most of your questions, except for weakness (2). Regarding this weakness, there are plenty of examples when an old technique provides a novel result,  or interesting insight in a new context. For example, low-rank factorizations are very old, but factorization machines were novel, and LORA training in language models is also novel. We believe that the combination with splines with factorization machines provides, on the one hand, interesting insight (segmentized, or "personalized" spline functions), while also being useful in practice for large scale real-time recommender systems.