Soboroff, Nicholas and Cahan recently proposed a method for evaluating the performance of retrieval systems without relevance judgments. They demonstrated that the system evaluations produced by their methodology are correlated with actual evaluations using relevance judgments in the TREC competition. In this work, we propose an explanation for this phenomenon. We devise a simple measure for quantifying the similarity of retrieval systems by assessing the similarity of their retrieved results. Then, given a collection of retrieval systems and their retrieved results, we use this measure to assess the average similarity of a system to the other systems in the collection. We demonstrate that evaluating retrieval systems according to average similarity yields results quite similar to the methodology proposed by Soboroff et al., and we further demonstrate that these two techniques are in fact highly correlated. Thus, the techniques are effectively evaluating and ranking retrieval systems by "popularity" as opposed to "performance." H.3.3 [Information Search and Retrieval]: Retrieval Models. General Terms: Theory, Experimentation. Keywords: Ranking Retrieval Systems. Soboroff, Nicholas and Cahan [2] recently proposed a method for evaluating retrieval systems in the absence of relevance judgments by (1) constructing a pool from the "top" documents retrieved by the underlying retrieval systems, (2) assigning relevance judgments at random within this pool, and (3) assessing the performance of the underlying systems using these artificially generated relevance judgments. A number of variants on this method for generating relevance judgments were investigated, including (1) constructing pools at varying depths, (2) viewing the pool of documents as sets versus multisets, and (3) assigning relevance judgments in varying ways within the pool. The resulting system assessments and rankings were well correlated with actual TREC rankings in their experiments. * This work partially supported by NSF Career Grant CCR-0093131 and NSF Grant 5-36955. As noted by Soboroff et al., these "pseudo-evaluations" suffered from a common phenomenon. While the bulk of the systems were classified correctly, the best (and therefore most important) systems were consistently ranked with the poor performers (see Figure 1) . Our hypothesis for this phenomenon is that these blind methods are suffering from a "tyranny of the masses" effect: the "better" systems are doing something significantly "different" from the more generic systems in the competition and, in the absence of actual relevance judgments, are being punished for this fact. Thus, we believe that the systems are, in fact, being evaluated in terms of "popularity" instead of "performance." In order to verify this hypothesis, we devise a simple measure for the similarity of two retrieval systems based on the similarity of their retrieved results, and we define the "popularity" of a system by its average similarity to the other systems in the collection. In experiments with TREC data, we demonstrate that assessing "system performance" by popularity yields results strikingly similar to those of the methods of Soboroff et al. Furthermore, we demonstrate that the Soboroff technique and the evaluation of systems by "popularity" are in fact highly correlated. Thus, these techniques are effectively evaluating and ranking retrieval systems by "popularity" as opposed to "performance."  RESULTS
A related method was suggested by Aslam and Savell REF who devised a measure for quantifying the similarity of the retrieval systems by assessing the similarity of their retrieval results.
On the effectiveness of evaluating retrieval systems in the absence of relevance judgments
SIGIR '03
Computer Science