on the robustness of relevance measures with incomplete judgments
We investigate the robustness of three widely used IR relevance measures for large data collections with incomplete judgments. The relevance measures we consider are the bpref measure introduced by Buckley and Voorhees [7], the inferred average precision (infAP) introduced by Aslam and Yilmaz [4], and the normalized discounted cumulative gain (NDCG) measure introduced by Jrvelin and Keklinen [8]. Our main results show that NDCG consistently performs better than both bpref and infAP. The experiments are performed on standard TREC datasets, under different levels of incompleteness of judgments, and using two different evaluation methods, namely, the Kendall correlation measures order between system rankings and pairwise statistical significance testing; the latter may be of independent interest.
robust relevance relevance measures measurement incomplete judgments data collection bpref inferred average precision precise infap normalized discounted cumulative gain ndcg experience standardization evaluation method correlation order systems statistics cranfield methodology information retrieval