Paper: Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity (ICLR-2024)

Reviewer comment (Novelty):
Change over SimCLR is incremental - the major weakness of SimCLR for non-ranking problems was complexity. The authors made a good point that hard negatives are clear for LTR (mentioned in S1), but SimCLR using small batches will likely largely resolve the issues. So the necessity for a new loss is not very convincing - in fact, the authors did not comprehensively compare with that baseline. Also, sometimes having easy negatives may improve the generalization of learning - this may need deeper study. SimSiam does not look to be a competitive baseline considering the factors. So the proposal of the new loss is not very convincing.

Author response:
**Will decreasing the number of negatives in SimCLR be sufficient to apply SimCLR to ranking problems?**

Thank you for asking this insightful question. First, we would like to clarify that our primary goal is to demonstrate that DL rankers built using pre-training techniques can achieve SOTA performance in LTR settings, and consequently beat GBDT models (prior SOTA). Only as a secondary goal, we investigate a few simple but effective pre-training strategies such as SimSiam, SimCLR, and our variant SimCLR-Rank. While we identified SimCLR-Rank has a positive impact in a wide variety of LTR scenarios, we also show that SimSiam works better for our large-scale proprietary dataset. Our goal is not to show that any one particular method is the best for pretraining in LTR.

As the reviewer correctly identified, SimCLR-Rank is different from SimCLR in two aspects: (1) it uses items from the same query group as freely available hard-negatives, (2) it reduces the number of negatives from batchsize to at most query group size and thus allow better computational scaling with batchsize. To identify which of these contribute more towards SimCLR-Rank’s performance, we performed an additional experiment below (as suggested by the reviewer), comparing SimCLR-Rank with a variant of SimCLR (which we call as SimCLR-sample). In SimCLR-sample, instead of using all the other items as negatives, we uniformly sample a constant number of negatives from the batch. This reduces the computational complexity from quadratic in batchsize (of SimCLR) to linear in batchsize. In the below experiments we follow the setup in Subsection 4.3.2 and SimCLR-Rank and SimCLR-sample use the same number of negatives and batchsize.

We find that SimCLR-Rank performs better on MSLR/Istella, while SimCLR-sample performs better on Yahoo. We note that MSLR/Istella are much sparser than Yahoo (see Table 12 in the paper), and are more representative of typical search and recommendations applications. Therefore we still believe that SimCLR-Rank which selects hard-negatives from the same query group is a useful pre-training method for the LTR toolbox and can be highly effective in many situations.

| Method | MSLR | Yahoo | Istella |
| --- | ----------- | - | - |
| SimCLR-Rank| **0.3929 $\pm$ 0.0018** | 0.5989 $\pm$ 0.0010 | **0.5830 $\pm$ 0.0013** |
| SimCLR-sample | 0.3890 $\pm$ 0.0008 | **0.6056 $\pm$ 0.0047** | 0.5787 $\pm$ 0.0035 |

Reviewer comment (Theory):
Extendability. The reviewer thought about this problem before and one major difference between tabular vs text is, tabular datasets assume a feature space, while text encoders are for text in general that is easy to generalize. For example, one can easily add more text fields and everything is still kind of in the “text space”, but adding new features to the tabular table may completely invalidate the pre-training in the previous feature space. So when the feature space changes, which can be common in practice, the pre-training needs to be done again. Also, the pre-training needs to be done for each dataset separately, unlike text domains where one model may be sufficient. Thus pre-trained models on tabular datasets may be much more limiting in general.

Author response:
**Pretrained models in tabular datasets are not very extendable.**

The reviewer’s observation that our pretrained models cannot take advantage of transfer learning is correct. Tabular datasets have specific features which can be very different. In fact, each tabular dataset is its own domain. Because in general there are no common features between tabular datasets, it is not clear how to have common knowledge. Levin et al. [1] show that if features are shared, it is possible to transfer knowledge between datasets, but in general it is not known how to do transfer learning in tabular data. The same shortcoming is true of GBDTs. Transfer learning for tabular data is a question of independent interest, and beyond the scope of our paper.

Reviewer comment (Evaluation):
It is unclear why no online results are provided for the industry dataset. Overall it is not clear how significant the results are in this section due to unclear baselines, private dataset, and unclear real-world implications.

Author response:
**It is not clear how significant the results are in the private dataset section.**

We agree with the reviewer that only online experiments can reveal the true impact of an LTR model due to the counterfactual nature of the LTR problem. That is from offline experiments, it is not clear how the user might have responded to the ranking provided by the model. However  running online experiments on real-world user traffic needs to be carefully considered as it has a potential opportunity cost. Unfortunately, due to superseding business priorities we could not run online experiments. Similarly, unfortunately at this point we cannot reveal more details of the baseline, but we will try our best to provide more details in the camera ready version. 

To provide more confidence in our private dataset results, we updated our Table 3 to provide error bars to our results.

| Target | Method | Delta% NDCG | Delta% Outlier NDCG |
| --- | ----------- | - | - |
| Purchase| Baseline | +0.00% | +0.00% |
|  | SimSiam | **+1.75% $\pm$ 0.13%** | **+29.66% $\pm$ 0.84%**|
| | SimCLR-Rank | +0.18% $\pm$ 0.13% | +2.19% $\pm$ 0.71% |
| Relevance| Baseline | +0.00% | +0.00% |
|  | SimSiam | **+2.78% $\pm$ 0.06%** | **+26.88% $\pm$ 0.35%**|
| | SimCLR-Rank | +0.85% $\pm$ 0.06% | +2.99% $\pm$ 0.31% |

Tabular (a.k.a.numerical) ML (e.g. LTR) systems are important for modern enterprises and therefore validating research hypotheses on large-scale practical datasets is highly valuable. Unfortunately, this is challenging due to the expensive and proprietary nature of such data and most LTR papers [2,3,4] do not prove their hypotheses on practical large-scale datasets. We demonstrate pretraining rankers achieve SOTA on public datasets but also emphatically on a large-scale commercial dataset. Therefore, we believe our results will be useful to the LTR community and the wider Tabular-DL community.

We thank the reviewer for their consideration, and are happy to help with any further concerns. If we have addressed the concerns we would appreciate it if the reviewer could consider raising their score.

[1] Transfer Learning with Deep Tabular Models. Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, Micah Goldblum. ICLR 2023.  
[2] Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?, Qin et al., ICLR 2021.  
[3] Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks., Ai et al., SIGIR 2019. 
[4] SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval., Pang et al., SIGIR 2020

Reviewer comment (Experiments):
This paper exhibits notable deficiencies in the aspects of experimental comparisons and discussions on related work. The experimental comparison methodology only considers comparisons between fine-tuning or probing methods based on pre-trained models, as well as MLP models. I believe that in the realm of learning to rank and tabular data, there are likely more recent deep learning methods that could serve as baselines for comparison. Proper discussions about these methods should also be incorporated into the related work section. Currently, I find the discussion on related work to be inadequate. This would aid in addressing certain evident issues, such as whether current methods for using deep learning to learn tabular data also employ contrastive learning for pre-training and what distinguishes them from the approach presented in this paper.

Author response:
Thank you for your review and feedback.  We would like to first take an opportunity to clarify the impact and contributions of our work.

We strongly believe our work will significantly impact industries that work with LTR problems (search, recommendation), by causing them to revise, or at least revisit, current SOTA methods. Today many large companies use GBDTs for ranking tabular data, and this approach is SOTA even in the research community. The novel starting point of our work is the often ignored fact that most of the data collected in LTR systems is unlabeled. We have shown through extensive experiments on standard public datasets and a large-scale private dataset that pretrained deep rankers can leverage unlabeled data to consistently outperform GBDTs, with outsized gains in robustness towards outlier data. 

All reviewers agree we have convincingly demonstrated this. Reviewer 1XYq: “Experimentally, authors show that pretrained models can outperform GBDTs, which is one of the strongest baselines, under label scarcity setting.” Reviewer oW1i: “the paper highlights the superior performance of pretrained deep rankers, especially on outlier queries, in scenarios with limited labeled data.” Reviewer i3NM: “The major contribution of the paper is to identify areas where [pretrained deep rankers] help, including query sparsity and label sparisity scenarios.”

We now address the concerns of the reviewer.

**The paper is missing important comparisons with other tabular SSL methods.**

We thank the reviewer for bringing this to our attention. We added a comprehensive and expanded related works section to Appendix A.1 (page 14 in the appendix, colored in blue) discussing the current state of pretraining for tabular (a.k.a numerical) dataset. After the paper decision, we plan to reorder the paper to place it into the main text.  

First, we would like to clarify that our primary goal is to demonstrate that DL rankers built using pre-training techniques can achieve SOTA performance in LTR settings, and consequently beat GBDT models (prior SOTA). Only as a secondary goal, we investigate a few simple but effective pre-training strategies such as SimSiam, SimCLR, and our variant SimCLR-Rank. While we identified SimCLR-Rank has a positive impact in a wide variety of LTR scenarios, we also show that SimSiam works better for our large-scale proprietary dataset.  Our goal is not to show that any one particular method is the best for pretraining in LTR.

SimCLR-Rank is different from SimCLR in two aspects: (1) it uses items from the same query group as freely available hard-negatives, (2) it reduces the number of negatives from batchsize to at most query group size and thus allow better computational scaling with batchsize. To identify which of these contribute more towards SimCLR-Rank’s performance, we performed an additional experiment below (as suggested by Reviewer i3NM), comparing SimCLR-Rank with a variant of SimCLR (which name SimCLR-sample). In SimCLR-sample, instead of using all the other items as negatives, we uniformly sample a constant number of negatives from the batch. This reduces the computational complexity from quadratic in batchsize (of SimCLR) to linear in batchsize. In the below experiments we follow the setup in Subsection 4.3.2 and SimCLR-Rank and SimCLR-sample use the same number of negatives and batchsize.

| Method | MSLR | Yahoo | Istella |
| --- | ----------- | - | - |
| SimCLR-Rank| **0.3929 $\pm$ 0.0018** | 0.5989 $\pm$ 0.0010 | **0.5830 $\pm$ 0.0013** |
| SimCLR-sample | 0.3890 $\pm$ 0.0008 | **0.6056 $\pm$ 0.0047** | 0.5787 $\pm$ 0.0035 |

As an additional comparison (suggested by reviewer 1XYq), we also evaluate SubTab [1] in the setting of Subsection 4.3.2.  SubTab’s pretraining objective divides input features into multiple subsets (in the language of computer vision, “multiple views”) and trains an autoencoder to reconstruct the original input features.  To pretrain using SubTab, we divide the input features into 4 subsets with 75% overlap, and with input corruptions of 15% masking probability and Gaussian noise of scale 0.1 (suggested in the SubTab paper as a good setting).  The finetuning strategy and the choice for encoder model is the same as for SimCLR-Rank and SimCLR-sample.  We find that SubTab does very poorly on the MSLR and Istella datasets while performing respectably on Yahoo (though worse than both SimCLR-Rank and SimCLR-Sample).

| Method | MSLR | Yahoo | Istella |
| --- | ----------- | - | - |
| SimCLR-Rank| **0.3929 $\pm$ 0.0018** | 0.5989 $\pm$ 0.0010 | **0.5830 $\pm$ 0.0013** |
| SimCLR-sample | 0.3890 $\pm$ 0.0008 | **0.6056 $\pm$ 0.0047** | 0.5787 $\pm$ 0.0035 |
|SubTab | 0.2879 $\pm$ 0.0019 | 0.5889 $\pm$ 0.0021 | 0.4700 $\pm$ 0.0031 |

Reviewer comment (Novelty):
Regarding the methodology, apart from applying SimCLR and SimSiam directly to pre-training for ranking tasks, the primary contribution of this paper is the SimCLR-Rank method. The key difference between this method and SimCLR lies in narrowing down the computation of contrastive learning from the entire batch to a single QG. This method possesses a degree of rationality and can effectively exploit the characteristics of the learning to rank task. However, in essence, this approach distills the semantic information contained in QG itself into the pre-trained model, similar to knowledge distillation (typically, QG is obtained through some form of retrieval, and the retrieval model can be viewed as a strong "teacher" model, while the pre-training model in this paper serves as a "student" model). Based on this observation, I believe that if the features from the retrieval model are input into GBDT, it could potentially achieve better performance, which is easily attainable in practical industrial applications. If this were the case, the application prospects of the method proposed in this paper would become rather limited.

Author response:
**Could we use features from the retrieval model to improve GBDT performance over SimCLR-Rank?**

We thank the reviewer for this very interesting conjecture.

First, we would like to note that the features of the retrieval model are in fact used in our LTR experiments on the private dataset. Despite this, SimCLR-Rank still outperforms the baseline model. Second, in the public datasets we do not have access to the upstream retrieval model features to test this hypothesis.

The last observation we would like to make is that retrieval models are typically not large or powerful in applications like search or recommendations. They are typically designed to be fast models (e.g. KNNs or even handcrafted rules) with high recall (and potentially low precision) to quickly filter millions or billions of items into hundreds of possibly relevant items. Since they are trained for recall and not for precision, these models and their features may find it difficult to differentiate and rank the relevant items in the query group. Therefore, we conjecture that  distilling from retrieval to rankers might not work well.

Reviewer comment (Novelty):
In SimCLR, dissimilar embeddings correspond to smaller weights, while similar embeddings correspond to larger weights, indicating its adaptive ability to mine hard samples. In the learning to rank scenario, samples from the same QG should be relatively similar, and samples from different QGs should be dissimilar. This implies that the original SimCLR method, even when calculating softmax over the entire batch, primarily emphasizes the samples from the same QG, aligning with the role of SimCLR-Rank. Therefore, I believe that the potential improvement that SimCLR-Rank can bring to typical ranking situations may be somewhat limited. I hope the authors can provide further explanations regarding this issue.

Author response:
**The potential improvement of SimCLR-Rank over SimCLR might be limited in practical situations.**

We thank the reviewer for their insightful question.  As the reviewer correctly identified, all the negative samples in the SimCLR-Rank loss are present in the SimCLR loss as well, so the qualitative and quantitative behaviors of the two algorithms are likely similar.  

However, SimCLR is difficult or impossible to use in large-scale big-data settings because of its high runtime and space complexity (which is quadratic in the batch size).  Reviewer i3NM also noted that SimCLR has this same shortcoming: “the major weakness of SimCLR for non-ranking problems was complexity.”  

To overcome this issue, we leverage the fact that in LTR the hard negatives are in the same query group (as the reviewer correctly notices), so we do not have to use the entire batch as negatives.  This (1) provides orders of magnitude improvement in speed (Table 6 in the paper, page 16), and (2) performs slightly better than SimCLR even with the runtime and memory savings (Table 5 in the paper, page 9).

To summarize, we do not claim that SimCLR-Rank’s loss will produce significantly different outcomes than SimCLR.  However, SimCLR-Rank is significantly faster, which makes it applicable in real-world big data settings for ranking.

We thank the reviewer for their consideration, and are happy to help with any further concerns. If we have addressed the concerns we would appreciate it if the reviewer could consider raising their score.

Reviewer comment (Novelty):
Technical novelty is limited.

Author response:
**Technical novelty is limited.**

We thank the reviewer for their feedback.  We would like to bring to the reviewer’s attention these following novel contributions:

1. We make the observation that in Learning-To-Rank (LTR) applications like search and recommendations, there is an abundance of unlabeled data. Before our work, this LTR problem setting has not been studied, to the best of our knowledge. **Reviewers agree on this point.** Reviewer 1XYq: “Representation learning in tabular LTR has not been studied much.” Reviewer oW1i: “The paper studies pre-trained DNN for tabular LTR problems, which is an underexplored problem.” Reviewer i3NM: “To the reviewer’s knowledge, this is the first work that shows some promises for pre-trained DNNs for the LTR task.”

2. SimCLR-Rank is the first pretraining method that specifically leverages LTR problem structure.  SimCLR, as noted by reviewer i3NM, is very slow: “the major weakness of SimCLR for non-ranking problems was complexity.”  Our method SimCLR-Rank leverages the unique structure of the LTR problem to gain orders of magnitude speed improvements (Table 6 on page 16 of the paper) while performing slightly better than SimCLR (Table 5 on page 9 of the paper).  We believe this is technically novel. 

3. We would finally like to emphasize that our results on the large-scale industry-size dataset are a big contribution. Most LTR papers [1,2,3] do not give results on industry-scale datasets, and do not prove their hypotheses on true large scale datasets because of lack of availability of such data and prohibitively large expense of running large-scale experiments. The fact that we demonstrate pretraining rankers achieve SOTA on public datasets in LTR but also **emphatically** on a large-scale commercial dataset is highly impactful. Therefore we believe our results will help the LTR and the wider Tabular-DL communities tremendously.

Reviewer comment (Experiments):
Datasets in experiments are limited to three datasets and one private dataset.

Author response:
**Datasets in experiments are limited to three datasets and one private dataset.**

We thank the reviewer for giving us an opportunity to clarify the experimental breadth. In LTR literature experiments these three datasets are generally considered the standard, with many influential and peer-reviewed work only evaluating at most the three public datasets we experiment on [1, 2, 3, 4]. Further, tabular (a.k.a. numerical) ML (e.g. LTR) systems are important for modern enterprises and therefore validating research hypotheses on large-scale practical datasets is valuable. Unfortunately, this is challenging due to the expensive and proprietary nature of such data. Given that we evaluate on all three of the standard public datasets—plus a large private dataset—we believe our empirical evaluation is stronger than comparable papers in the literature and could be valuable for the vast LTR and Tabular-DL communities.

Reviewer comment (Experiments):
Representation learning on tabular data not necessarily needs to consider LTR setting, since multi-layer MLP will be finetuned for LTR task. As a paper that proposes a new tabular self-supervised learning method, it lacks the comparison with other existing methods, such as  
- Hajiramezanali, E., Diamant, N. L., Scalia, G., & Shen, M. W. (2022, October). STab: Self-supervised Learning for Tabular Data. In *NeurIPS 2022 First Table Representation Workshop*.  
- Wang, W., KIM, B. H., & Ganapathi, V. (2022). RegCLR: A Self-Supervised Framework for Tabular Representation Learning in the Wild.  
- Ucar, T., Hajiramezanali, E., & Edwards, L. (2021). Subtab: Subsetting features of tabular data for self-supervised representation learning. *Advances in Neural Information Processing Systems*

Author response:
**Lacking comparison and discussion with existing tabular SSL methods.**

We thank the reviewer for bringing up the comparison with existing tabular SSL methods.  We add a comprehensive and expanded related works section to section A.1 (in the appendix of the paper, colored in blue) discussing the current state of tabular SSL.  After the paper decision, we plan to reorder the paper to place it into the main text. 

We would like to clarify that our primary goal is to demonstrate that DL rankers built using pre-training techniques can achieve SOTA performance in LTR settings, and consequently beat GBDT models (prior SOTA). Only as a secondary goal, we investigate a few simple but effective pre-training strategies such as SimSiam, SimCLR, and our variant SimCLR-Rank. While we identified SimCLR-Rank has a positive impact in a wide variety of LTR scenarios, we also show that SimSiam works better for our large-scale proprietary dataset.  Our goal is not to show that any one particular method is the best for pretraining in LTR.

Regarding the suggested tabular SSL methods: STab [12] applies a SimSiam-like method to tabular self-supervised learning, RegCLR [13] aims to extract tables from images and is not applicable to our setting, and SubTab [14] is a well-known and influential tabular SSL technique.  Given that STab has a similar strategy as SimSiam which we already evaluate in our paper (and because there was no published code for STab), we focus on evaluating SubTab in this response (respecting the time constraints of the rebuttal). 

We evaluate SubTab [14] in the setting of Subsection 4.3.2.  SubTab’s pretraining objective divides input features into multiple subsets (in the language of computer vision, “multiple views”) and trains an autoencoder to reconstruct the original input features.  To pretrain using SubTab, we divide the input features into 4 subsets with 75% overlap, and with input corruptions of 15% masking probability and Gaussian noise of scale 0.1 (suggested in the SubTab paper as a good setting).  The finetuning strategy and the choice for encoder model is the same as for SimCLR-Rank.  We find that SubTab performs respectably on Yahoo, but fails to learn the MSLR and Istella datasets.

| Method | MSLR | Yahoo | Istella |
| --- | ----------- | --- | ---- |
| SimCLR-Rank | **0.3929 $\pm$ 0.0018** | **0.5989 $\pm$ 0.0010** | **0.5830 $\pm$ 0.0013** |
| SubTab | 0.2879 $\pm$ 0.0019 | 0.5889 $\pm$ 0.0021 | 0.4700 $\pm$ 0.0031 |

Reviewer comment (Evaluation):
Why does pretraining improve outlier NDCG?

Author response:
**Why does pretraining improve outlier NDCG?**

It is a well-known phenomenon that pretraining helps improve robustness and out of distribution performance in domains like image and text [5, 6]. The same phenomenon, we find, is also true in the tabular LTR setting. Liu et al., [7] find that pretraining with data augmentation allows models to learn richer features they can generalize to outlier or out of distribution examples better–merely supervised training only incentivizes models to attend to highly predictive features in the training set.  We conjecture that pretraining with augmentations in LTR (as we do in our paper) improves performance on outlier NDCG for similar reasons.

Reviewer comment (Experiments):
GBDT could use some well-known semi-supervised learning methods such as pseudo-labeling or consistency regularization to use unlabeled dataset. Do the pretraining methods still outperform GBDT with semi-supervised learning?

Author response:
**Do the pretraining methods still outperform GBDT with semi-supervised learning?**

We thank the reviewer for bringing up important baselines. Rubachev et al. [9] find that consistency regularization does not help GBDTs (https://openreview.net/forum?id=kjPLodRa0n&noteId=E4FT6VCltG). As suggested by the reviewer, we ran new experiments on pseudolabeling for GBDTs, and found that it (typically) performs worse than the GBDT baselines themselves. 

Methodology: we allow 0.2% of the query groups to be labeled in each dataset, we train a GBDT on the labeled set and then use the GBDT to label the unlabeled train set.

A GBDT ranker outputs real-valued scores. Given that pseudolabeling datasets using GBDTs in LTR is not a well-studied problem, we propose the following approach. To convert real-valued scores into relevance scores, we take the lowest output score when labeling the unlabeled train set, and subtract it from all real scores. Then we round these positive floats into their nearest integers, and let these be the pseudolabeled relevance scores. In MSLR, this results in the pseudolabels being integers from 0-7, in Istella 0-12. For Yahoo we increase the spread of the scores by multiplying the positive floats by 100 before rounding to the nearest integer (resulting in the pseudolabels being numbers from 0-5), since without doing this all the pseudolabels would be 0.

We find that overall, pseudolabeling decreases the performance of GBDTs on MSLR and Istella (the sparser datasets which are typically more representative of search and recommendations applications, check Table 10 in the paper) while slightly improving the performance for Yahoo, which is somewhat an “easy” dataset. All the GBDT results below underperform pretrained deep models (we use the methodology in Section 4.1.1).  Note that due to the size of the dataset, the GBDT results are deterministic (no stderr).

| Method |MSLR | Yahoo | Istella |
| --- | ----------- | - | - |
| GBDT |0.3908|0.5932|0.5807|
| GBDT + pseudo-labeling| 0.3773 | 0.5967 | 0.5396|
| Pretrained DL | **0.3959 $\pm$ 0.0022** | **0.6107 $\pm$ 0.0035** | **0.5839 $\pm$ 0.0049** |

We thank the reviewer for their consideration, and are happy to help with any further concerns. If we have addressed the concerns we would appreciate it if the reviewer could raise their score.