SCONER: Scoring Negative Candidates Before Training Neural Re-Ranker For Question Answering
Abstract
A neural re-ranker aims to re-scores a set of candidates given by a search engine.
It is crucial to obtain good performance on many down-stream tasks such as retrieval-based question answering (ReQA).
In this work, we introduce a scoring function for negative candidates to train a neural re-ranker and compare models trained by our approach with three baselines on a range of
ReQA tasks.
We term our approach as SCONER— scoring negative candidates before training neural re-ranker, which includes 1) a scoring function based on the concept of Semantic Textual Similarity (STS) and data augmentation; and 2) a neural re-ranker trained on data using generated negativeness scores as labels.
Experimental results show that SCONER outperforms three baselines by up to 13% absolute improvement on the SearchQA dataset and 5.5% on average across all datasets in terms of P@1.
SCONER demonstrates that using different negativeness scores to train a neuralranker is better than a single score, and we present a simple yet efficient way to generate the scores.
1. Introduction
Due to the wide applications in real-world, Retrieval Based
Question Answering (ReQA has gained increasing interest and attention in recent years and many benchmarks have been proposed for Retrieval Based Question Answering (ReQA)
(Cohen et al., 2018;
Khot et al., 2020; Ahmad et al., 2019; Guo et al., 2021).
A promising approach for solving ReQA involves two stages: (1) retrieve a small set of candidates from a large corpus and (2) re-rank these candidates.
The re-ranking stage can significantly improve the initial retrieval performance (Ozyurt et al., 2020) and thus it is crucial for any retrieval system (Ma et al., 2021). .edu > .
ICML workshop on Knowledge Retrieval and Language Mod- els, Baltimore, Maryland, USA, 2022.
Copyright 2022 by the author(s).
[Table] Large Pretrained Language Models (PrLMs) have been widely used as neural re-rankers (Yilmaz et al., 2019;
Nogueira & Cho, 2019) 1 .
In most cases, the negative examples used to train the re-ranker are assigned with the same label.
However, we argue that some candidates may be more negative than others and should be treated differently.
Figure 1 shows an example from HotpotQA dataset (Yang et al., 2018) to illustrate this argument.
In this example, neither S1 nor S2 contains the correct answer; yet S1 mentions a key entity in the question (David Beckham), while S has no common entity with the question.
From the human perspective, S1 should have a higher score than S2.
It leads us to ask a question - “is having different levels of negativeness beneficial for training neural re-rankers?”
Driven by this question, we propose an approach for scoring negative candidates (§2).
Our approach has two stages.
First, we train a model on STS benchmark (Conneau &
Kiela, 2018).
This model generates a high score for a twosentence pair if they are semantically similar; otherwise, a low score.
Second, we use this STS model to generate scores for the question and negative candidate pairs.
In this way, we obtain a set of question-candidate pairs with labels in the continuous range of [0 , 5] as opposed to previous works where labels are binary.
Furthermore, we want the generated score for a negative candidate to be higher than others if the first candidate has more relevant information to the answer.
To achieve this goal, we explore three data augmentation techniques (§2.2).
Such scoring approach allows: 1) a good candidate that is not annotated as an answer to have a high score, 2) more negative samples to be used to train a neural re-ranker, and 3) negative candidates to be differentiated using “negativeness” scores.
In this paper, negativeness score means the score for a question and a negative candidate; and a higher score means the negative candidate contains more information to answer the question.
We compare three standard training strategies and our proposed method on the MultiReQA
(Guo et al., 2021) benchmark, which includes five training datasets across different domains.
Based on our experiments, we observe that 1) our proposed approach outperforms three baselines by up to 13% absolute improvement on the SearchQA dataset and 5.5% on average across all datasets in terms of P@1; 2) use of a different negativeness score achieves better performance than the same score even when fewer negative candidates are used; and 3) our proposed method has a significant advantage in a low resource setting.
These lead to the answer to the question that use of a negativeness score is an efficient way to train a neural re-ranker.
2. Negative Candidate Scoring Approach
The key idea of the negative candidate scoring approach is to utilize a Semantic Textual Similarity (STS) model and the motivation is that STS score determines how close two sentences are in terms of semantic meaning (Conneau & Kiela, 2018)
(see Appendix B for details).
In the following, we describe the two stages of our scoring approach: (1) training an STS model, and (2) using it to generate negativeness scores for the question and negative candidate pairs.
2.1. Training an STS-model
We train an STS model on the STS benchmark, which is a regression model consisting of a RoBERTa model (Liu et al., 2019) and a Multi-Layer Perceptron (MLP) layer.
In particular, the input to the RoBERTa model is [CLS] sentence1 [SEP] sentence2 [SEP] .
Then we feed the representation of the [CLS] token to the MLP layer which predicts a score (see the figure of the model’s input and output in Appendix C).
Mean Squared Error (MSE) loss is taken as the training objective to minimize the gap between the predicted score with the ground truth STS score.
2.2. Negativeness Score Generation
We use the STS model to generate scores for the question and negative candidate pairs.
Due to the fact that sometimes, the important information is only presented in the answer but not in the question, even though a candidate is relevant to a question, the STS model might not produce a high score.
To overcome this issue, we introduce three ways to augment a question to consider the answer in the scoring process.
We expect that if two candidates have similar information regarding a question, but one has more similar information to the answer than the other, then the first one should obtain a higher score.
Next, we present each augmentation approach (examples can be found in Table 5 in Appendix D).
Question + Answer (Q+A)
The first approach is to simply concatenate the answer to the original question.
Question + Keywords of Answer (Q+KA)
The second approach is to extract the keywords from the answer and concatenate the keywords to the original question.
We use
Rapid Automatic Keyword Extraction (RAKE) (Rose et al., 2010) to extract the keywords.
We believe that answer might include irrelevant information and it can be removed By extracting keywords.
Neglecting irrelevant information can help the STS model generate a reasonable negative score.
Keywords of Question and Answer (KQ+KA)
This method extracts the keywords not only for the answer but also the question.
We concatenate the keywords sequentially.
The intuition is the same as the second approach but also applies to the question.
3. SCONER: A Neural re-ranker
Generated negative scores are used in training our model, i.e., SCONER.
The model has the identical model architecture as the STS model (§2.1) but with different inputs, i.e., the inputs of the STS model are a pair of sentences from the STS-benchmark, while the inputs of SCONER are question-candidate pairs.
3.1. Training Pipeline
The pipeline to train SCONER consists of three steps (the left part in Figure 2).
Step1 (the green block) : we use BM to retrieve the top-100 candidates for a question.
From these candidates, we further randomly sample 10 negative candidates.
Step2 (the blue block) : we augment the question by one of the approaches described in §2.2 and use it as sentence1 and each negative candidate from step1 is used as sentence2 .
We feed the sentence1 and sentence to the STS model and obtain a score for the question and negative candidate pair.
Step3 (the yellow block) : we train
SCONER using the question and positive candidate pairs which are given in the training set and the question and negative candidate pairs which are given in step 1.
We use
MSE loss to optimize our model.
Since the upper bound of a negative score is 5 given by the STS model, we assign 5 to all positive candidates as a positive candidate should have a higher score than any negative candidate. [Figure]
3.2. Inference Pipeline
During the inference time, for any given question, we retrieve the top-100 candidates using BM25.
We then concatenate the question with every candidate and ask SCONER to predict a score for each candidate.
Finally, we re-rank the top-100 candidates based on the predicted scores and select top-k candidates as the final answer.
4. Experiments and Results 4.1. Dataset and Baselines
We conduct experiments on MultiReQA benchmark (Guo et al., 2021) which includes five training datasets:
SearchQA (SQA) (Dunn et al., 2017), TriviaQA (TQA) (Joshi et al., 2017), HotpotQA (HQA) (Yang et al., 2018), SQuAD(Rajpurkar et al., 2016), and NaturalQuestions(NQ)(Kwiatkowski et al., 2019).
More details and the statistics of each dataset are given in Appendix E.
We compare our proposed approach with three commonly used neural model baselines: Binary Classification Model (BCM), Regression Model (RM), and Triplet Model (TM).
More details about baselines can be found in Appendix F.
All baseline models and our proposed models have the same size and are based on the RoBERTa base pretrained model.
Experimental setup can be found in Appendix G.
4.2. Results and Analysis
We use two standard metrics to evaluate each model defined by MultiReQA, P@1 and MRR (see Appendix H).
In the following, we mainly describe P@1, however, it is easy to see the same trend extended to MRR.
Comparison with Baselines Table 1 shows the SCONER outperforms all baselines across all datasets.
The largest gain SCONER achieved is ∼ 13%, compared to BCM on
SearchQA, and the largest average gain is ∼ 5.5%, compared to RM.
While compare to the strongest baseline, i.e.,
TM (since TM outperforms other two baselines), SCONER achives ∼ 2.5%, ∼ 4%, ∼ 3%, and ∼ 5% improvement in terms of P@1 on NQ, SQuAD, HotpotQA, and SearchQA, respectively, and outperforms TM on TriviaQA by a small margin.
This shows that using more negative candidates and differentiating the negative candidates are important to boost the models’ performance.
Comparison with Existing Methods The existing methods on MultiReQA directly retrieve answers from the entire corpus without re-ranking.
We present one term-matching (e.g. BM25) and two neural-retrieval based methods from
Guo et al.
(2021), which are fine-tuned BERT dual encoder and USE-QA (Yang et al., 2020) on each in-domain dataset.
Other baselines and our model re-rank candidates after BM25 retrieval.
From the results, we see that the reranking phase improves the performance significantly.
For instance, re-ranking improve P@1 at least ∼ 20%, ∼ 13%, ∼ 42%, ∼ 38%, and ∼ 20% on NQ, SQuAD, HotpotQA,
SearchQA and TriviaQA, respectively, compared to BM25.
Effect of Data Augmentation We also train neural rerankers with scores generated by the STS model without augmentation (Q model) to see the effect of augmentation.
From Table 1, we can see that baselines outperform Q model in most cases.
For example, the performance of Q models is worse than TM on NQ, SearchQA and TriviaQA.
Moreover,
BCM and TM outperforms the Q model on average.
On the other hand, using augmentation methods, Q+A, Q+KA,
[Table]
and KQ+KA are better than Q models and outperform all baselines on average.
While three proposed data augmentation all outperform baselines, KQ+KA is the best technique not only because it achieves the best average score but also because it performs more stable across all datasets.
This demonstrates that it is important to incorporate the answer to the question in the negative score generation process.
Effect of Size of Negative Candidates Here, we compare using 1/3/5/7/10 negative candidates per question to train
MultiReQA-SQuAD SCONER.
Figure 3 shows the trend of
P@1 score with different numbers of negative candidates for each method.
We see that for three out of four methods, 5 or 7 already yields best performance, which suggests that
SCONER does not need to be trained with many negative candidates.
In addition, we have two observations: 1) compared to 1 negative candidate per question, 5/7/10 always yield better performance, and this suggests that using one negative candidate is not enough to train SCONER; 2) compared to the TM baseline, which uses 10 negative candidates per question, all four models perform better than TM even though using less negative candidates (e.g. 3 and 5), this demonstrates that using different negativeness scores is an effective way to train a neural re-ranker.
Effect of Size of Training Data We use 5/10/15/18 thousands (K) questions to train our models and the baselines on the SQuAD dataset.
For TM and our models, each question is paired with 10 negative candidates.
Figure 4 illustrates
[Equation]
From
Figure 4, we see that (1) except for TM, the other two baselines and our SCONER get improvement as the training size increases, (2) SCONER and TM perform well even with small amount of training data (e.g. 5K), but BCM and RM are much worse, and (3) SCONER always performs better than BCM and RM, and the advantage of SCONER is more significant in low resource settings.
5. Ablation Study
We present most insightful studies in this section and other ablation studies can be found in Appendix I.
5.1. What are the Effects of STS Model?
STS scores can be used to approximate the scores for question and candidates because STS and question-candidate ranking are related so that these two tasks require similar knowledge or skill to solve.
To further justify this intuition, we use the STS model to initialize a re-ranker rather than using the RoBERTa weights.
We expect to see that the STS model will be better than a RoBERTa
[Table]
model.
We repeat the experiments in Table 1 but use the
STS model rather than RoBERTa model to initialize each neural re-ranker and present the results in Table 2.
We use green/red color to represent improvements/decrements compared to Table 1 (deeper color means more significant improvements/decrements).
From Table 2, we can see that the STS model is better than the RoBERTa model in most cases, which justifies our intuition and to some extent explains why the proposed score generation approach can improve the model performance.
5.2. Can SCONER be Applied to Positive Candidates?
In our previous pipeline, SCONER only uses the generated scores for negative candidates.
Here, we also use the STS model to generate scores for positive candidates and use them to train re-ranker instead of using fixed score 5 as other experiments mentioned previously.
We test on the
SQuAD dataset.
Table 3 shows the performance of each SCONER trained with generated scores or fix score 5, and the best baselines model on SQuAD dataset which is RM.
We see that except for Q model, other three SCONERs are all better than the best baseline model, this suggests that the score for the positive candidates can be used in the training time.
However, we also see that using score 5 is better than the generated scores in all cases.
We further find that for the best SCONER model, Q+KA, 98% of the negative candidates have lower scores than the corresponding positive candidates, but the average generated score for the positive candidates is 3.64, which is less than 5.
This suggests that a larger gap between scores of positive and negative candidates helps the model to differentiate the positive and negative candidates better.
In addition, we also observe that the performance of the Q model, which does not use any augmentation in the generation, is much worse than the score 5 while the other three methods are similar.
This suggests that the scores generated by augmentation are more reliable.
[Table]
6. Limitations
In this paper, we study the retrieval-based question answering task and propose a new training strategy for a crossattention re-ranker model.
While we compare with three standard baselines and two simple retrievers, recently, there are many interesting neural retrievers have been proposed such as DPR (Karpukhin et al., 2020), ACNE (Xiong et al., 2020), SPARTA (Zhao et al., 2021), ColBERT-QA (Khattab et al., 2021), and Poly-DPR (Luo et al., 2022).
Comparing SCONER with these latest neural retrievers will be an interesting future work.
7. Conclusion
Most previous training approaches for ReQA use the same labels for all negative candidates, we argue that different candidates should have different negativeness scores based on their semantic relevance to the question.
Motivated by this, we ask the question - “can a neural re-ranker yield better performance trained on different negativeness scores?”.
To answer this question, we present SCONER, a new pipeline to train neural re-rankers by generating scores for negative candidates which is based on the semantic meaning between question-candidate pairs.
Our experimental results show that SCONER outperforms all standard training methods across five datasets and demonstrate that using negativeness scores to train a neural re-ranker is better than using the same labels.
Our proposed method makes negative candidates differentiable which further allows us to use more negative samples to train neural re-ranker.
A. Related Work A.1. Retrieval Based Question Answering
ReQA is to identify sentences from large corpus that contain the answer to a question (Yang et al., 2015; Cakaloglu et al., 2020; Ahmad et al., 2019; Guo et al., 2021).
It has practical applications such as s Googles Talk to Books 2 .
ReQA is similar to Open Domain Question Answering (ODQA) but different in the following aspect, ReQA aims to build an efficient retrieval system, and the answer is a sentence or a short passage (Ahmad et al., 2019); while ODQA requires a retrieval system to find relevant documents at a large scale and a machine reading comprehension model to predict short answer span from documents (Bilotti et al., 2007; Chen & Van Durme, 2017;
Chen et al., 2017; Min et al., 2019; Karpukhin et al., 2020).
In this paper, we focus on the ReQA task and believe that building an efficient system for ReQA is also beneficial for the
ODQA task.
For example, QASC (Khot et al., 2020) requires retrieving sentences from a large corpus and composing them to answer a multiple-choice question, and a good ReQA system can be used to retrieve sentences in the first stage.
A.2. Neural Re-Ranker
Bag-of-words ranking models such as BM25 (Robertson & Zaragoza, 2009) have a long history in information retrieval.
Although being efficient, these methods depend on handcrafted features and can not be optimized for a specific task such as ReQA.
Therefore, a re-ranker is trained on a down-stream task to re-score the candidates after the first step retrieval.
Neural networks have been applied as re-rankers (Guo et al., 2016;
Hui et al., 2017; Xiong et al., 2017; Dai et al., 2018;
McDonald et al., 2018), also called as answer selection models in some work (Rao et al., 2016; Yang et al., 2015; Rao et al., 2019; Laskar et al., 2020).
Boosting technique has been proposed to train a neural re-ranker where the training samples are assigned with different weights(Makino & Iwakura, 2017).
Recently, large language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are widely used as re-rankers (Nogueira & Cho, 2019;
Yilmaz et al., 2019; MacAvaney et al., 2019).
Such re-rankers take the concatenation of a query and a candidate as input and apply attention technique(Vaswani et al., 2017) to allow rich interaction between the question and the candidate.
Then a classification or regression module (scoring layer) is added on top to compute a score.
Binary classification entropy (BCE) is usually used to train a re-ranker, but BCE has limitations such as a large number of negative candidates being unused to create balanced training samples.
Triplet loss addresses this issue by the idea of learning to rank (Liu, 2009).
However, none of these methods addresses the concern whether we can use different negativeness scores to train a neural re-ranker.
Similar to previous work, we use large
PrLMs as re-rankers but different from theirs, we train a model using different scores for negative candidates.
Re-ranking using ensemble models have been explored recently (Zhang et al., 2021), but since their systems are more complex than ours in terms of model size, we don’t compare with them.
B. Sentence Textual Similarity (STS)
Review STS STS determines how close two sentences are in terms of semantic meaning (Conneau & Kiela, 2018).
Specifically, given two sentences, a high STS score indicates that they present similar meanings; while a low score implies that they have different meanings.
The STS score is in the range of [0, 5].
Table 4 shows two pairs of sentences with score 0 and 5 from the STS-B dataset.
Score 5 means two sentences are semantically equivalent and score 0 means semantically irrelevant.
[Table]
A man is playing a guitar. A man plays the guitar.
[Table]
A young man is playing the piano. [Table] peeling
[Table]
prawn [Table] different meaning. Motivation of Using STS to Generate Scores STS lays the foundation of our scoring approach because there is a relation between the STS task and the question-candidate ranking task.
Considering a question and a candidate pair, if the candidate has similar information regarding the question, then it is likely to be a relevant candidate (corresponding to a high STS score); on the contrary, if it has less similar information, then it is likely to be irrelevant (corresponding to a lower STS score).
Meanwhile, STS is better than other methods of finding similar information because it considers the semantic meaning of two sentences such as synonyms of words.
C. Figure of STS-Model and Neural Re-ranker
Figure 5 shows the architecture of the models used in all experiments, except for binary classification model where the output of MLP is two logits.
Our model consists of two parts.
A RoBERTa model takes the concatenation of two sentences as input and output the contextual representation of [CLS] token.
An MLP layer takes this representation as input and output a score.
BERT
CLS
[Figure]
s
[Figure]
s...
s n
SEP cc... c n
SEP
CLS
[Figure]
' s 2 ' .
[Figure]
[Figure]
[Figure]
[Figure]
D. Data Augmentation
While STS and our question-answer sentence are similar task in some sense, there is a difference between identifying semantic similarity and answer ranking.
The former focuses on detecting meaning equivalence and the latter emphasizes keyword matching and semantic understanding.
Thus, even though a candidate is relevant to a question, the STS model might not produce a high score due to the semantic meaning gaps between the candidate and the question.
To illustrate this, consider a question “Beyonce has a fan base that is referred to as what?” and a good candidate “The name Bey Hive derives from the word beehive, purposely misspelt to resemble her first name, and was penned by fans after petitions on the online social networking service Twitter and online news reports during competitions” (the answer is Bey Hive ), the STS model generate a low score for this pair as shown in the first row (Q) of Table 5.
[Table]
[Figure] [Table] [Table] [Table]
[Table]
E. Data Processing and Statistics
The dataset includes two parts, question-answer pairs and a corpus.
The question-answer pairs are from MRQA (Fisch et al., 2019).
MRQA is a collection of extractive QA task where the goal is to extract an answer span given a question and a context.
The corpus is given by MultiReQA.
Particularly, to convert extractive QA task to ReQA task, where the context is not given, Guo et al.
(2020) divide the context into single sentences and combine all sentence to construct a corpus.
The goal is to retrieve the answer to a question from the corpus.
We refer reader to see the details of processing of the corpus in
Guo et al.
We remove all questions that do not have any answer in the corpus in both training and testing sets.
Table 6 shows the number of questions, the number of candidates (the size of the corpus) and the average number of answers per questions of each dataset.
The average number of answers for SearchQA and TriviaQA are more than others.
Notice that the number of questions in the training time might be different than the MRQA dataset since we only use those samples where the answer sentences are not empty.
Questions Domain and Reasoning Type Table 7 shows the domains and reasoning type of each dataset.
We see that
the
MultiReQA include datasets from different domains and different types of reasoning skill required to answer the question.
F. Baselines
Binary Classification Model (BCM)
We use the RoBERTa model as the encoder, which takes input as [CLS] question [SEP] candidate [SEP] .
Then, we feed the vector representation of [CLS] to a linear layer with two logits as outputs: one represents the probability of candidates being irrelevant and the other represents it being relevant.
We apply binary cross entropy loss to train this model.
The training data is constructed by using the positive samples for each question with label 1, and we randomly selected the same amount of negative samples from the top-100 candidates given by BM25 with label 0.
Regression Model (RM)
This baseline is similar to the BCM baseline, but the linear layer only outputs one logit instead of two, thus it is a regression model rather than a binary classification model.
We use MSE loss to train this model.
The positive and negative samples are the same as BCM, but the positive samples have label 5.
We also use 1 as the label for positive samples but find that 5 yields better performance, thus we use label 5 to train all RM baselines.
Appendix I.1 shows the results comparison between labels as 1 and 5.
Triplet Model (TM)
This baseline has the identical model architecture as the RM baseline, but we use the triplet loss to train the model and in this way, more negative candidates can be used.
Specifically, each training sample is a triplet, i.e., ⟨ q, c + , c − ⟩ , where q is a question, c
+ is a positive candidate, and c − is a negative candidate.
Let S ( q, c ) denote the score given by the model for question q and candidate c .
The model is trained such that S ( q, c + ) is higher than S ( q, c − ) .
We use the same negative candidates to train TM and SCONER, but SCONER use generated score as labels.
[Table]
G. Experiment Setup
We use Huggingface (Wolf et al., 2020) and Pytorch (Paszke et al., 2019) implementation for training each model.
For each dataset, we use up to 25 thousands questions to train a neural re-ranker.
To train the STS model, we use one GTX
GPU with maximum length (MaxL) 128, batch size (bs) 32, learning rate (lr) 2e-5, and 6 training epochs (epochs).
For each neural re-ranker (including the baseline), we use four GTX1080 GPUs with MaxL 368, bs 16, lr 2e-5, 5 epochs, and gradient accumulation steps 2.
H. Evaluation Metrics
We present two evaluation metrics as follows.
Precision@K P@K reveals the proportion of top-K retrieved candidates that are relevant.
R@K reveals the proportion of relevant documents are in the top-K retrieved candidates.
In Eq 1, N is the number of questions, A K are the top-K retrieved answer, A ∗ is correct answers.
[Equation]
Mean Average Precision P@K does not take the position of relevant candidates into account, which means a system that ranks the relevant answer higher than another system can not be identified as better.
MAP address this issue, computed as follows, where in Eq 2, Rel @
i is 1 if the i th answer is correct, 0 otherwise.
[Equation]
MRR
The MRR score is computed as follows, MRR
[Equation]
where rank i is the rank of the first relevant answer.
I. Ablation Study I.1. Ablation Study for Regression Model Baseline
Here, we explore two labels for regression models, 0 and 5 on three datasets.
We train each model by initializing it with
RoBERTa.
Table 8 shows the results, and we found that label 5 is better than 0 in three cases.
I.2. How Many Candidates are Needed for Re-ranking?
To answer this question, we use 50/100/150/200 candidates in the re-ranking time.
We test on three datasets using the MRR metric and consider the best model of each dataset given in Table 1 (P@1 in Table 10), and they are KQ+KA, Q+KA, and
Q+A models for NQ, SQuAD, and HQA, respectively.
From Table 9, we find that the performance gap of SQuAD and NQ are more noticeable than of HotpotQA.
For SQuAD, re-ranking 200 candidates yields ∼ 1 % improvement compared to 50, and for NQ, re-ranking 100 candidates yields ∼ 1 % improvement compared to 50.
But for HotpotQA, re-ranking 50 candidates surprisingly yields the best performance, ∼ 0 .
5% better than 200.
Further investigation reveals that the recall of BM25 on HotpotQA is already 99% for 50 candidates and increasing the size of candidates rather introduces more distracting candidates in the re-ranking time; but for SQuAD and
NQ, the recall of BM25 increases.
On the other hand, re-ranking more candidates causes longer inference time, i.e. the inference time of re-ranking 50/100/150/200 candidates is 0.49/0.85/1.24/1.63 seconds.
This suggests that if the recall of less candidates is already high enough (e.g. 99% ), then using less candidates in re-ranking is time efficient and gets best performance.
[Table]
[Table]