  Learning to Rank (LTR) algorithms are usually evaluated using Information
Retrieval metrics like Normalised Discounted Cumulative Gain (NDCG) or Mean
Average Precision. As these metrics rely on sorting predicted items' scores
(and thus, on items' ranks), their derivatives are either undefined or zero
everywhere. This makes them unsuitable for gradient-based optimisation, which
is the usual method of learning appropriate scoring functions. Commonly used
LTR loss functions are only loosely related to the evaluation metrics, causing
a mismatch between the optimisation objective and the evaluation criterion. In
this paper, we address this mismatch by proposing NeuralNDCG, a novel
differentiable approximation to NDCG. Since NDCG relies on the
non-differentiable sorting operator, we obtain NeuralNDCG by relaxing that
operator using NeuralSort, a differentiable approximation of sorting. As a
result, we obtain a new ranking loss function which is an arbitrarily accurate
approximation to the evaluation metric, thus closing the gap between the
training and the evaluation of LTR models. We introduce two variants of the
proposed loss function. Finally, the empirical evaluation shows that our
proposed method outperforms previous work aimed at direct optimisation of NDCG
and is competitive with the state-of-the-art methods.
