On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search (2021)

Abstract: According to the Probability Ranking Principle (PRP), ranking documents in decreasing order of their probability of relevance leads to an optimal document ranking for ad-hoc retrieval. The PRP holds when two conditions are met: [C1] the models are well calibrated, and, [C2] the probabilities of relevance are reported with certainty. We know however that deep neural networks (DNNs) are often not well calibrated and have several sources of uncertainty, and thus [C1] and [C2] might not be satisfied by neural rankers. Given the success of neural Learning to Rank (LTR) approaches—and here, especially BERT-based approaches—we first analyze under which circumstances deterministic neural rankers are calibrated for conversational search problems. Then, motivated by our findings we use two techniques to model the uncertainty of neural rankers leading to the proposed stochastic rankers, which output a predictive distribution of relevance as opposed to point estimates. Our experimental results on the ad-hoc retrieval task of conversation response ranking reveal that (i) BERT-based rankers are not robustly calibrated and that stochastic BERT-based rankers yield better calibration; and (ii) uncertainty estimation is beneficial for both risk-aware neural ranking, i.e. taking into account the uncertainty when ranking documents, and for predicting unanswerable conversational contexts.

## introduction according to the probability ranking principle (prp) @xcite , ranking documents in decreasing order of their probability of relevance leads to an optimal document ranking for ad-hoc retrieval 2 . @xcite discussed that for the prp to hold, ranking models must at least meet the following conditions: [c1] assign well calibrated probabilities of relevance, i.e. if we gather all documents for which the model predicts relevance with a probability of e.g. 30%, the amount of relevant documents should be 30%, and [c2] report certain predictions, i.e. only point estimates such as, for example, 80% probability of relevance. dnns have been shown to outperform classic information retrieval (ir) ranking models over the past few years in setups where considerable training data is available. it has been shown that dnns are not well calibrated in the context of computer vision @xcite . if the same is true for neural l2r models for ir, e.g. transformer-based models for ranking @xcite , [c1] is not met. additionally, there are a number of sources of uncertainty in the training process of neural networks @xcite that make it unreasonable to assume that neural ranking models fulfill @xcite : parameter uncertainty (different combinations of weights that explain the data equally well), structural uncertainty (which neural architecture to system for documents that are likely relevant @xcite . in this paper we first analyze the calibration of neural rankers, specifically bert-based rankers for ir tasks to evaluate how calibrated they are. then, to model the uncertainty of bert-based rankers, we propose stochastic neural ranking models (see figure 1 ), by applying different techniques to model the uncertainty of dnns, namely mc dropout @xcite and deep ensembles @xcite which are agnostic to the particular dnn. in our experiments, we test models under distributional shift, i.e. the test data distribution is different from the training data, also referred to as out-ofdistribution (ood) examples @xcite . in real-world settings, there are often inputs that are shifted due to factors such as non-stationarity and sample bias. additionally, this experimental setup provides a way of measuring whether the dnn "knows what it knows" @xcite , e.g. by outputting high uncertainty for ood examples. we find that bert-based rankers are not robustly calibrated. stochastic bert-based rankers have 14% less calibration error on average than bert-based rankers. uncertainty estimation from stochastic bert-based rankers is advantageous for downstream applications as shown by our experiments for risk-aware neural ranking (2% more effective on average relative to a model without risk-awareness) and for predicting unanswerable conversational contexts (improves classification by 33% on average of all conditions). ## related work calibration and uncertainty in ir even though to optimally rank documents according to the prp @xcite requires the model to be calibrated @xcite ([c1]), the calibration of ranking models has received little attention in ir. in contrast, in the machine learning community there have been a num3 in this paper we use risk and uncertainty interchangeably. the second condition ([c2]) for optimal retrieval when ranking according to the prp @xcite is that models report predictions with certainty. while the (un)certainty has not been studied in neural l2r models, there are classic approaches in ir that model the uncertainty. such approaches have been mostly inspired by economics theory, treating variance as a measure of uncertainty @xcite . following such ideas, nonneural ranking models that take uncertainty into account (i.e. risk-aware models), and thus do not follow the prp @xcite , have been proposed @xcite , showing significant effectiveness improvements compared to the models that do not model uncertainty. uncertainty estimation is a difficult task that has other applications in ir besides improving the ranking effectiveness: it can be employed to decide between asking clarifying questions and providing a potential answer in conversational search @xcite ; to perform dynamic query reformulation @xcite for queries where the intent is uncertain; and to predict questions with no correct answers @xcite . ## method in this section we introduce the methods used for answering the following research questions: rq1 how calibrated are deterministic and stochastic bert-based rankers? rq2 are the uncertainty estimates from stochastic bert-based rankers useful for risk-aware ranking? rq3 are the uncertainty estimates obtained from stochastic bertbased rankers useful for identifying unanswerable queries? we first describe how to measure the calibration of neural rankers ([c1]), followed by our approach for modeling and ranking under uncertainty ([c2]), and then we describe how we evaluate their robustness to distributional shift. ## experimental setup we consider three large-scale information-seeking conversation datasets foot_3 that allow the training of neural ranking models for conversation response ranking: msdialog @xcite ) contains 246k context-response pairs, built from 35.5k information seeking conversations from the microsoft answer community, a qa forum for several microsoft products; mantis @xcite ) contains 1.3 million context-response pairs built from conversations of 14 stack exchange sites, such as askubuntu and travel; udc dstc8 @xcite ) contains 184k context-response pairs of disentangled ubuntu irc dialogues. ## conclusions in this work we study the calibration and uncertainty estimation of neural rankers, specifically bert-based rankers. we first show that the deterministic bert-based ranker is not robustly calibrated for the task of conversation response ranking and we improve its calibration with two techniques to estimate uncertainty through stochastic neural ranking. we also show the benefits of estimating uncertainty using risk-aware neural ranking and for predicting unanswerable conversational contexts. as future work, investigating the use of stochastic rankers in other settings is important, such as other neural l2r architectures, other search and retrieval tasks @xcite @xcite , and the ensembling of neural rankers @xcite .