Controlling the complexity in comparing search user interfaces via user studies
Over time, researchers have acknowledged the importance of understanding the users strategies in the design of search systems. However, when involving users in the comparison of search systems, methodological challenges still exist as researchers are pondering on how to handle the variability that human participants bring to the comparisons. This paper present methods for controlling the complexity of user-centered evaluations of search user interfaces through within-subjects designs, balanced task sets, time limitations, pre-formulated queries, cached result pages, and through limiting the users access to result documents. Additionally, we will present our experiences in using three measures  search speed, qualified search speed, and immediate accuracy  to facilitate the comparison of different search systems over studies.
search user interfaces methodology user-centered evaluation evaluation measures