Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance (2025)

Abstract: Although synthetic data has changed various aspects of information retrieval (IR) pipelines, the main training paradigm remains: contrastive learning with binary relevance labels, where one positive document is compared against several negatives using the InfoNCE loss. This objective treats all documents that are not explicitly annotated as relevant on an equally negative footing, regardless of their actual degree of relevance, thus missing subtle nuances useful for ranking. To overcome this limitation, in this work, we forgo real documents and annotations and use large language models to directly generate synthetic documents that answer the MS MARCO queries according to _several different levels of relevance_. We also propose using Wasserstein distance as a more effective loss function for training transformer-based retrievers with graduated relevance labels. Our experiments on MS MARCO and BEIR benchmark show that our proposed approach outperforms conventional training with InfoNCE by a large margin. Without using any real documents, our method significantly improves self-supervised retrievers and is more robust to distribution shift compared to contrastive learning using real data. Our method also successfully integrates existing real data into the synthetic ranking context, further boosting the performance. Overall, we show that generating multi-level ranking contexts is a better approach to synthetic data generation for IR than just generating the standard positive and negative documents.

## introduction the ability of information retrieval (ir) methods to rank a collection of documents based on their relevance to a given query is critical for many applications like web search and, more recently, retrieval * equal contributions. ## related work dense retrieval training retrieval training pipelines have improved significantly by addressing various limitations of ir datasets. to better delineate the positive and negative regions, many have proposed using a large number of random in-batch negatives @xcite and similarly using existing retrievers to mine hard-todetect negatives for each query @xcite @xcite . some have also used existing retrievers to filter out the unannotated positives during hard negative mining @xcite . however, the fundamental limitation remains: binary relevance labels provide a crude approximation of the ranking task. ## synthetic ranking context for list-wise training (sycl) to better approximate the inference objective, we propose to train dense retrievers on passages with multiple levels of relevance, thus creating a rich multi-level ranking context for each query. since most large-scale ir datasets only provide passages with binary ground truth labels, we use llms to generate passages with graduated relevance levels for each query (section 3.1). additionally, we propose to use wasserstein distance as a loss function that uses graded relevance labels and multiple positives per query more effectively than alternative list-wise losses (section 3.2). ## experiments our experiments demonstrate the effectiveness of synthetic multi-level ranking contexts and the wasserstein loss for training dense retrievers. first, without using any real documents or annotations, sycl fine-tuning improves the performance of selfsupervised dense retrievers. second, we show that the wasserstein loss with multiple levels of relevance outperforms infonce using the same queries and passages. third, we find that sycl training only on synthetic documents performs similarly to contrastive training with real data of the same size on trec dl, while on average, it outperforms it in terms of out-of-domain generalization on beir. overall, we achieve the best ranking effectiveness when incorporating existing real data into our synthetic multi-level ranking context. through additional analytical experiments, we show the individual impact of the wasserstein loss and graduated relevance labels. finally, we inspect different components of our data generation pipeline and find that even smaller, 32b-scale llms can generate high-quality data comparable to larger 70bparameter models. ## additional analysis fine-grained relevance levels are necessary for achieving good performance. to separate the impact of using multiple relevance levels from the wasserstein loss, we repeat our main experiment with the wasserstein loss but instead use binary labels. we assign relevance levels 1 and 0 to more relevant (levels 3 and 2) and less relevant (levels 1 and 0) synthetic passages, respectively ( even with the same data and loss function, finegrained relevance levels are necessary for achieving good performance: using binary relevance levels instead decreases the boost in performance by 4.0 ndcg@10 on average across all sets. although our main comparison is between binary and multi-level synthetic data, we also experiment with fine-tuning on real binary data using wasserstein loss ( wasserstein loss is more effective than other list-wise loss functions. we compare our proposed wasserstein loss against other list-wise loss functions that can take advantage of multiple levels of relevance ( the advent of dense retrieval. we also evaluate the kl divergence, which is often used for model distillation but has also been used for training with a multi-level ranking context @xcite . except for ranknet, all other loss functions take advantage of multiple levels of relevance and outperform the binary infonce loss. however, the wasserstein loss is the most effective and provides significant gains over the next best loss function (listnet). generating multi-level synthetic data is better even for binary training. in our experiments thus far, we approximate binary synthetic data by using the same multi-level synthetic passages but converting the labels from multi-level to binary. this helps us study the impact of label granularity without confounding factors like variation in passage content. we now evaluate directly generating binary data using qwen 2.5 32b and report the results (ndcg@10) in even with a strict interpretation of relevance, models trained with sycl outperform bm25 without using any real passages. we show that even under a strict interpretation of relevance labels, our method outperforms bm25 without using any real passages or their annotations ( ## conclusion in this work, we introduce sycl, a novel method that first uses llms to generate rich multi-level ranking contexts and then the wasserstein distance to train retrievers with multiple levels of relevance. we show that llms can successfully generate synthetic data with graduated relevance levels, significantly improving the effectiveness of unsupervised retrievers. when using the same synthetic queries and passages, sycl utilizes the available data more effectively and performs better than training with binary relevance labels. sycl can also combine real and synthetic datasets to further improve performance. moreover, we show that wasserstein distance is more effective at fine-tuning transformerbased retrievers with graduated relevance labels and performs better than the usual list-wise loss functions. our results show that generating multiple passages with graduated relevance levels is a better approach to synthetic data generation for ir than generating the standard positive and negative passages. these results encourage future work to explore synthetic data generation methods that are better suited for information retrieval tasks, going beyond the binary definition of relevance.