Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLM s (2025)

Abstract: Training robust retrieval and reranker models typically relies on large-scale retrieval datasets; for example, the BGE collection contains 1.6 million query-passage pairs sourced from various data sources. However, we find that certain datasets can negatively impact model effectiveness — pruning 8 out of 15 datasets from the BGE collection, reduces the training set size by 2.35×, surprisingly increases nDCG@10 on BEIR by 1.0 point. This motivates a deeper examination of training data quality, with a particular focus on “false negatives”, where relevant passages are incorrectly labeled as irrelevant. We utilize LLMs as a simple, cost-effective approach to identify and relabel false negatives in training datasets. Experimental results show that relabeling false negatives as true positives improves both E5 (base) and Qwen2.5-7B retrieval models by 0.7-1.4 points on BEIR and by 1.7-1.8 points at nDCG@10 on zero-shot AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of LLMs to identify false negatives is supported by human annotation results. Our training dataset and code are publicly available.

## introduction modern-day retrievers and rerankers are datahungry, relying on large and high-quality training datasets to accurately retrieve or rerank on challenging domains @xcite @xcite . a typical training dataset for information retrieval (ir) has multiple instances consisting a training query, labeled positive passages, and a set of mined hard negative passages. sampling hard negatives has been consistently used in retrieval models to improve downstream retrieval accuracy @xcite @xcite . ## the rlhn methodology in this section, we discuss the llm judge cascading framework, training dataset modifications, and dataset postprocessing and statistics. ## analysis ablation on hard-negatives and significance tests. as an ablation, we experiment with the number of hard negatives during fine-tuning e5 (base) in tevatron. from robustness of rlhn across varying training data subsets. as training datasets can be large, relabeling all training pairs using the llm cascading pipeline can be computationally prohibitive. from figure 5 , we demonstrate that rlhn remains robust and maintains similar accuracy gains, even when applied to smaller randomly sampled subsets of the training dataset. to evaluate this, we use four random subsets (100k, 250k, 400k, and 680k) of the training datasets, with each dataset's distribution shown in air-bench (avg. 5) figure 5 : ndcg@10 scores on beir (avg. 16 and avg. 7) and air-bench (avg. 5) by fine-tuning e5 (base) on a subset of the 100k, 250k, 400k, and 680k training pairs using the "rlhn" technique for both stages. all individual dataset scores for both beir and air-bench are provided in figure 7 and figure 8 . overall, we have two main findings: (i) the e5 (base) model fine-tuned on rhln stages 1 and 2 training data, with false hard negatives relabeled as positives, consistently outperforms the default setting, and (ii) the steeper slope in ndcg@10 demonstrates continual improvement across zeroshot domains, as the amount of training data increases, especially as observed in air-bench. reranker distillation is competitive but limited in detecting false negatives. a reranker, or crossencoder, is commonly used in knowledge distillation to fine-tune a retriever model as an alternative to the traditional contrastive or infonce loss function @xcite @xcite . this approach bypasses the original relevance judgments, relying instead on knowledge encoded within the reranker itself. rather than using rlhn, we evaluate how well rerankers detect false negatives in training datasets. specifically, we rerank the hard negatives for each training instance and compute two metrics: (i) map@10, which measures the average precision of false negatives in the top-10 results, and (ii) p@l(gt), which measures the precision of false negatives among the top-k results, where k equals the number of false negatives. ## human validation we conducted a validation study with three human assessors conducting using label studio foot_9 for data annotation. the assessors were briefed on the relevance task, and then independently evaluated a total of 670 query-hard negative pairs. the hard negatives were randomly sampled from the rlhn set, each containing at least one false negative. during the assessment, all annotators worked independently and were not exposed to the llm query ground truth or positive passages false negatives (detected by rlhn) (q1) which is a food magazine, latin mass magazine or saveur? latin mass magazine: a journal of catholic culture, commonly referred to as latin mass magazine, is an american catholic magazine published quarterly, with a traditionalist catholic viewpoint. [ ... ] saveur: saveur is a gourmet, food, wine, and travel magazine that specializes in essays about various world cuisines. its slogan-"savor a world of authentic cuisine"-signals the publication's focus on enduring culinary traditions [ ... ] food & wine: food & wine is a monthly magazine published by time inc. it was founded in 1978 by ariane and michael batterberry. it features recipes, cooking tips, travel information, restaurant reviews, chefs, wine pairings and seasonal content [ ... ] cocina (magazine): is a colombian-based monthly magazine published by publicaciones semana s.a.. it features recipes, cooking tips, culinary tourism information, restaurant reviews, chefs, wine pairings and seasonal holiday content [ ... ] (q2) what year was the premier professional ice hockey league in the world established? yin and yang: in chinese philosophy, yin and yang (also, yin-yang or yin yang) describes how apparently opposite or contrary forces are actually complementary, interconnected, and interdependent in the natural world, and how they give rise to each other as they interrelate to one another. yin and yang: yin and yang are ancient chinese philosophical terms, with the yin yang theory being a fundamental part of feng shui. it is a chinese theory on the perspective of continuous change and balance. [ ... ] yin yang symbols and their meanings: in a nutshell, chinese yin yang symbols represent perfect balance. a great deal of chinese philosophy stems from the concept of yin and yang -opposites interacting [ ... ] (q4) charles, prince of wales is patron of numerous other organizations. charles, prince of wales: charles, prince of wales (born 14 november 1948) is the eldest child and heir apparent of queen elizabeth ii [ ... ] charles's interests encompass a range of humanitarian and social issues: he founded the prince's trust in 1976, sponsors the prince's charities, and is patron of numerous other charitable and arts organisations. [ ... ] julia cleverdon dame: julia charity cleverdon [ ... ] served for 16 years as chief executive of business in the community, one of the prince's charities of charles, prince of wales. the prince's trust: the prince's trust is a charity in the united kingdom founded in 1976 by charles, prince of wales, and frederick john pervin to help young people. [ ... ] ## qualitative analysis of false negatives we qualitatively analyze the labeling accuracy of our llm cascading framework by manually spotchecking a few training instances. as shown in ## conclusion in this work, we emphasize the importance of clean training datasets. first, we showed that certain datasets can negatively impact model effectiveness when fine-tuned across a huge collection with many training pairs. dataset pruning removes 57.5% (8 datasets out of 15) and improves the model accuracy on beir by even 1.0 point and making the dataset 2.35× smaller. next, after pruning, we observed the issue of false hard negatives in the remaining training datasets, where passages in the hard negative list are misclassified and are relevant to the query. we presented rlhn, an effective cascading llm approach for relabeling hard negatives as ground truth or positives. using rlhn, both retrievers and rerankers consistently improved their model generalization on beir and zero-shot air-bench evaluations, as supported by human annotation results, outperforming competitive baselines such as hard-negative sampling and cross-encoder knowledge distillation.