  Recommendation systems are often evaluated based on user's interactions that
were collected from an existing, already deployed recommendation system. In
this situation, users only provide feedback on the exposed items and they may
not leave feedback on other items since they have not been exposed to them by
the deployed system. As a result, the collected feedback dataset that is used
to evaluate a new model is influenced by the deployed system, as a form of
closed loop feedback. In this paper, we show that the typical offline
evaluation of recommender systems suffers from the so-called Simpson's paradox.
Simpson's paradox is the name given to a phenomenon observed when a significant
trend appears in several different sub-populations of observational data but
disappears or is even reversed when these sub-populations are combined
together. Our in-depth experiments based on stratified sampling reveal that a
very small minority of items that are frequently exposed by the deployed system
plays a confounding factor in the offline evaluation of recommendation systems.
In addition, we propose a novel evaluation methodology that takes into account
the confounder, i.e the deployed system's characteristics. Using the relative
comparison of many recommendation models as in the typical offline evaluation
of recommender systems, and based on the Kendall rank correlation coefficient,
we show that our proposed evaluation methodology exhibits statistically
significant improvements of 14% and 40% on the examined open loop datasets
(Yahoo! and Coat), respectively, in reflecting the true ranking of systems with
an open loop (randomised) evaluation in comparison to the standard evaluation.
