  According to the Probability Ranking Principle (PRP), ranking documents in
decreasing order of their probability of relevance leads to an optimal document
ranking for ad-hoc retrieval. The PRP holds when two conditions are met: [C1]
the models are well calibrated, and, [C2] the probabilities of relevance are
reported with certainty. We know however that deep neural networks (DNNs) are
often not well calibrated and have several sources of uncertainty, and thus
[C1] and [C2] might not be satisfied by neural rankers. Given the success of
neural Learning to Rank (L2R) approaches-and here, especially BERT-based
approaches-we first analyze under which circumstances deterministic, i.e.
outputs point estimates, neural rankers are calibrated. Then, motivated by our
findings we use two techniques to model the uncertainty of neural rankers
leading to the proposed stochastic rankers, which output a predictive
distribution of relevance as opposed to point estimates. Our experimental
results on the ad-hoc retrieval task of conversation response ranking reveal
that (i) BERT-based rankers are not robustly calibrated and that stochastic
BERT-based rankers yield better calibration; and (ii) uncertainty estimation is
beneficial for both risk-aware neural ranking, i.e.taking into account the
uncertainty when ranking documents, and for predicting unanswerable
conversational contexts.
