Abstract As retrieval systems become more complex, learning to rank approaches are being developed to automatically tune their parameters. Using online learning to rank, retrieval systems can learn directly from implicit feedback inferred from user interactions. In such an online setting, algorithms must obtain feedback for effective learning while simultaneously utilizing what has already been learned to produce high quality results. We formulate this challenge as an exploration-exploitation dilemma and propose two methods for addressing it. By adding mechanisms for balancing exploration and exploitation during learning, each method extends a state-of-the-art learning to rank method, one based on listwise learning and the other on pairwise learning. Using a recently developed simulation framework that allows assessment of online performance, we empirically evaluate both methods. Our results show that balancing exploration and exploitation can substantially and significantly improve the online retrieval performance of both listwise and pairwise approaches. In addition, the results demonstrate that such a balance affects the two approaches in different ways, especially when user feedback is noisy, yielding new insights relevant to making online learning to rank effective in practice. Keywords Information retrieval Á Learning to rank Á Implicit feedback An earlier version of this article appeared in Hofmann et al. (2011a) . In this substantially revised and extended version we introduce a novel approach for balancing exploration and exploitation that works with pairwise online learning approaches, and carefully evaluate this new approach. Comparisons with the earlier described algorithm for listwise approaches yield new insights into the behavior of the two types of approach in online settings, especially how they compare in the face of noisy feedback and how they react to a balance of exploration and exploitation.
Besides listwise online learning to rank approaches for IR, a pairwise approach was investigated in REF .
Balancing exploration and exploitation in listwise and pairwise online learning to rank for information retrieval
Information Retrieval
Information Retrieval
Computer Science