on retrieval tests with an inhomogeneous query collection
Retrieval tests are often performed with an inhomogeneous query collection, that means certain properties of queries vary within the collection. This paper investigates under which conditions this inhomogeneity may lead to a loss of measurement precision and validity and should be controlled by an experimental design. In particular, interrelationships between the design of a retrieval test, inhomogeneity of the query collection and measurement theoretical properties of the evaluation measure will be discussed. Based on these results, some recommendations for experimental design of retrieval tests are presented.
retrieval test queries queries collect paper measurement precise validation experimental design design evaluation recommendation query