Reciprocal Rank Fusion (RRF), a simple method for combining the document rankings from multiple IR systems, consistently yields better results than any individual system, and better results than the standard method Condorcet Fuse. This result is demonstrated by using RRF to combine the results of several TREC experiments, and to build a meta-learner that ranks the LETOR 3 dataset better than any previously reported method.  Categories and Subject While supervised learning-to-rank methods have garnered much attention of late, unsupervised methods are attractive because they require no training examples. In the search for such a method we came up with Reciprocal Rank Fusion (RRF) to serve as a baseline. We found that RRF, when used to combine the results of IR methods (including learning to rank), almost invariably improved on the best of the combined results. We also found that RRF consistently equaled or bettered other methods we tried, including established metaranking standards Condorcet Fuse and CombMNZ (cf. [4]). RRF simply sorts the documents according to a naive scoring formula. Given a set D of documents to be ranked and a set of rankings R, each a permutation on 1..|D|, we compute where k = 60 was fixed during a pilot investigation and not altered during subsequent validation. Our intuition in choosing this formula derived from fact that while highlyranked documents are more important, the importance of lower-ranked documents does not vanish as it would were, say, an exponential function used. uments according to the pairwise relation r(d1) < r(d2), which is determined for each (d1, d2) by majority vote among the input rankings. CombMNZ requires for each r a corresponding scoring function sr : D → R and a cutoff rank c which all contribute to the CombMNZ score: We conducted four pilot experiments, each combining the results of 30 configurations of Wumpus Search applied to four different TREC collections. The results of the first, shown in table 1, indicated that k = 60 was near-optimal, but that the choice was not critical. The results also showed, somewhat unexpectedly, that RRF bested competing approaches, as well as more sophisticated learning methods whose investigation was the original impetus for our work. We repeated our experiment with four sets of submissions to TREC tasks; the particular sets were selected because they have been used in previous metaranking evaluation. It is worthy of note that, while our pilot runs used exactly the same set of Wumpus configurations to generate the individual rankings on different datasets, the individual rankings in these experiments were exactly those submitted by TREC participants. Table 2 shows the RRF result, as well as the best individual, Condorcet and CombMNZ results. The MAP score for RRF exceeds that of Condorcet Fuse in all cases, and CombMNZ in all but one. RRF also outperforms the best ranking in each experiment, with the exception of TREC 9, where the best ranking was derived using a humanin-the-loop. RRF outperforms the next-best ranking, which was automated. The pilot and TREC experiments indicate that RRF outperforms Condorcet, CombMNZ and the best system by 4% to 5% on average. We use a simple sign test to establish significance. Discounting the first pilot run, RRF outperformed Condorcet all 7 times (p ≈ 0.008), outperformed CombMNZ 6 of 7 times (p ≈ .04), and outperformed the best individual result either 6 or 7 times (0.008 ≤ p ≤ 0.04), depending on whether or not the manual result is considered. Thus all measured differences are significant. Our final experiment used the sample learning results supplied with the LETOR 3 1 dataset, as well as a logistic gradient descent method (LGD) which we are developing. For the purpose of analysis, we combined the seven sets of documentquery pairs into one and computed an overall MAP score. 1 research.microsoft.com/enus/um/beijing/projects/letor 758
REF proposed RRF (Reciprocal Rank Fusion) scores for combining different rank lists in information retrieval.
Reciprocal rank fusion outperforms condorcet and individual rank learning methods
SIGIR
Computer Science