Pre-trained language models (PLMs), e.g., BERT (Devlin et al., 2019) , ELECTRA (Clark et al., 2020) and T5 (Raffel et al., 2019) , have achieved state-of-the-art results on standard ad-hoc retrieval benchmarks and in many NLP tasks. The success of PLMs mainly relies on learning contextualized representations of input sequences using the Transformer (Vaswani et al., 2017) . The Transformer uses a self-attention mechanism whose computational complexity is quadratic with respect to the input sequence's length, so PLMs generally limit the sequence's length (e.g., to 512 tokens) to reduce computational costs. Consequently, when applied to the ad-hoc ranking task, PLMs are commonly used to predict the relevance of passages or individual sentences. (Dai and Callan, 2019b; . The max or k-max passage scores (e.g., top 3) are then aggregated to produce a document relevance score. Such approaches have achieved state-of-the-art results on a variety of ad-hoc retrieval benchmarks.
Documents are often much longer than a single passage, however, and intuitively there are many types of relevance signals that can only be observed in a full document. For example, the Verbosity Hypothesis (Robertson and Walker, 1994) states that relevant excerpts can appear at different positions in a document. It is not necessarily possible to account for all such excerpts by considering only the top passages. Similarly, the ordering of passages itself may affect a document's relevance; a document with relevant information at the beginning is intuitively more useful than a document with the information at the end . On the other hand, the amount of non-relevant information in a document can also be a signal, because relevant excerpts would make up a large fraction of an ideal document. IR Axioms encode this idea in the first length normalization constraint (LNC1), which states that adding non-relevant information to a document should decrease its score (Fang et al., 2011) . Considering a full document as input has the potential to incorporate signals like these. Furthermore, from the perspective of training a supervised ranking model, the common practice of applying document-level relevance labels to individual passages is undesirable, because it introduces unnecessary noise into the training process.
Empirical studies support the importance of fulldocument signals. Wu et al. study how passagelevel relevance labels correspond to document-level labels, finding that more relevant documents also contain a higher number of relevant passages . Additionally, experiments in several works suggest that aggregating passage-level relevance scores to predict the document's relevance score outperforms the common practice of using the maximum passage's score (Bendersky and Kur-land, 2008; Fan et al., 2018; Ai et al., 2018) .
In this work, we study how PLMs like BERT can be applied to the ad-hoc document ranking task while preserving many document-level signals. To this end, we propose PARADE, an end-to-end document reranking model. PARADE predicts a document's relevance by learning passage-level relevance representations that are aggregated in a way that preserves document-level context. These aggregation approaches include 1) a passage weighting method, 2) a pooling technique, and 3) using the Transformer in a hierarchical way. PARADE is optimized end-to-end at the document level, which eliminates the noise introduced by using the document relevance label as a proxy for passage relevance labels. Since the utilization of full-text causes more memory usage, we investigate using knowledge distillation to create smaller, more efficient PARADE models that remain effective.
In the recent TREC-COVID challenge that studies the problem of identifying literature relevant to COVID-19 information needs, PARADE performed well and was among the top positions in the second round (as measured by nDCG@10). The details of our TREC-COVID submissions can be found in Appendix A.1.
Our contributions are threefold:
• The proposal of the end-to-end PARADE method for predicting a document's relevance by aggregating passage representations, • An evaluation on standard TREC ad-hoc benchmark collections confirming the effectiveness of our approach, and • Analyses of how PARADE's efficiency can be improved by decreasing the model size, and of how its effectiveness is influenced by the number of passages considered and by the initial ranking method.
We review three lines of related research. Contextualized Language Models for IR. Neural IR models like DSSM (Huang et al., 2013) , DRMM (Guo et al., 2016) , (Co-)PACRR (Hui et al., 2017 , and (Conv-)KNRM (Xiong et al., 2017; Dai et al., 2018) have been proposed for the ad-hoc retrieval task. However, their contextual capacity is limited by using pre-trained unigram embeddings. Benefiting from BERT's pre-trained contextual embeddings, BERT-based IR models have been shown to be superior to neural IR mod- (Dai and Callan, 2019b) explore the sentence-level and passage-level relevance signals using BERT for document reranking, respectively. CEDR proposed a joint approach that combines BERTs outputs with existing neural IR models (MacAvaney et al., 2019) . Other researchers trade off PLM effectiveness for efficiency by utilizing the PLM to improve document indexing Dai and Callan, 2019a) , pre-computing intermediate Transformer representations (Khattab and Zaharia, 2020; MacAvaney et al., 2020a; Gao et al., 2020; Humeau et al., 2020) , using the PLM to build sparse representations (MacAvaney et al., 2020b) , or reducing the number of Transformer layers (Hofstätter et al., 2020b,a) .
While several works have recently investigated approaches for improving the Transformer's efficiency by reducing the computational complexity of its attention module, none of these approaches have been shown to be effective for the document ranking task. The Sparse Transformer (Child et al., 2019) and Reformer (Kitaev et al., 2020) focus on text generation. We were unable to effectively use Transformer-XL (Dai et al., 2019) in pilot experiments, while Longformer (Beltagy et al., 2020) is an interesting contemporaneous work. We note that PARADE could be used in conjunction with such models.
Passage-based Document Retrieval. Given the increasing lengths of documents in full-text collections, Callan first experimented with paragraphbased and window-based methods of defining passages (Callan, 1994) . Several works drive passagebased document retrieval in the language modeling context (Liu and Croft, 2002; Bendersky and Kurland, 2008) , indexing context (Lin, 2009) , and learning to rank context (Sheetrit et al., 2020) . In the realm of neural networks, HiNT demonstrated that aggregating representations of passage level relevance can perform well in the context of pre-BERT models (Fan et al., 2018) . Others have investigated sophisticated evidence aggregation approaches (Zhao et al., 2020; . Wu et al. explicitly modeled the importance of passages based on position decay, passage length, length with position decay, exact match, etc . In a contemporaneous study, they proposed a model that considers passage-level representations of relevance in order to predict the passage-level cumulative gain of each passage (Wu et al., 2020) . In this approach the final passage's cumulative gain can be used as the document-level cumulative gain. Our approaches share some similarities, but theirs differs in that they use passagelevel labels to train their model and perform passage representation aggregation using a LSTM.
Knowledge Distillation. Knowledge distillation is the process of transferring knowledge from a large model to a smaller student model (Ba and Caruana, 2014; Hinton et al., 2015) . Ideally, the student model performs well while consisting of fewer parameters. One line of research investigates the use of specific distilling objectives for intermediate layers in the BERT model (Jiao et al., 2019; . Turc et al. pre-train a family of compact BERT models and explore transferring task knowledge from large fine-tuned models (Turc et al., 2019) . Tang et al. distill knowledge from the BERT model into Bi-LSTM (Tang et al., 2019) . Tahami et al. propose a new cross-encoder architecture and transfer knowledge from this model to a bi-encoder model for fast retrieval (Tahami et al., 2020) . We demonstrate this approach can be applied to PARADE to improve efficiency without substantial reductions in effectiveness.
In this section, we present the proposed PARADE method for end-to-end document reranking. Given a query q and a document D, a ranking method aims to generate a relevance score rel(q, D) that estimates to what degree document D satisfies the query q. As described in the following sections, we perform this relevance estimation by aggregating passage-level relevance representations into a document-level representation, which is then used to produce a relevance score. Representing a Document as Passages. As introduced in Section 1, a long document cannot be considered directly by the BERT model due to its fixed sequence length limitation. Following (Dai and Callan, 2019b; Callan, 1994) , we split a document into passages that can be handled by BERT individually. To do so, a sliding window of 150 words is applied to the document with a stride of 100 words, formally expressed as D = {P 1 , . . . , P n } where n is the number of passages. Afterward, these passages are taken as input to the BERT model for relevance estimation.
Creating Passage Relevance Representations. Following prior work (Nogueira and Cho, 2019), we concatenate a pair of query q and passage P i with a [SEP] token in between and another [SEP] token at the end. The special [CLS] token is also prepended, in which the corresponding output in the last layer is parameterized as a relevance representation p cls i ∈ R d , denoted as follows:
Aggregating Passage Relevance Representations. Given the passage relevance representations D cls = {p cls 1 , . . . , p cls n }, PARADE summarizes D cls into a single dense representation d cls ∈ R d in three different ways, coined as PARADE Max , PARADE Attn , and PARADE.
PARADE Max utilizes a robust max-pooling operation on the passage relevance features 1 in D cls . As widely applied in Convolution Neural Network, max-pooling has been shown to be effective in obtaining position-invariant features (Scherer et al., 2010) . Herein, each element at index j in d cls is obtained by a element-wise max-pooling operation on the passage relevance representations over the same index.
PARADE Attn assumes that each passage contributes differently to the relevance of a document to the query. A simple yet effective way to learn the importance of a passage is to apply a feed-forward network to predict passage weights:
where softmax is the normalization function and W ∈ R d is a learnable weight. For completeness of study, we also introduce a PARADE Avg that simply averages the passage relevance representations. This can be regarded as manually assigning equal weights to all passages (i.e., w i = 1/n). PARADE Transformer , which as shorthand we simply call PARADE, enables passage relevance representations to interact by adopting the Transformer (Vaswani et al., 2017) in a hierarchical way. Specifically, BERT's [CLS] 2 token embedding and all p cls i are concatenated, resulting in an input x l = (emb cls , p cls 1 , . . . , p cls n ) that is consumed by Transformer layers to exploit the ordering of and dependencies among passages. That is,
where LayerNorm is the layer-wise normalization as introduced in (Ba et al., 2016) , MultiHead is the multi-head self-attention (Vaswani et al., 2017) , and FFN is a two-layer feed-forward network with a ReLu activation in between. The [CLS] vector of the last Transformer output layer, regarded as a pooled representation of the relevance between query and the whole document, is taken as d cls . The sequence length of the Transformer layers in PARADE is equal to the number of passages used in a document, usually dozens, hence this approach adds only a small amount of computation compared with PARADE Attn and PARADE Max . Generating the Relevance Score. For all three PARADE variants, after obtaining the final d cls embedding, a single-layer feed-forward network is adopted to generate a relevance score, as follows:
where W d ∈ R d is a learnable weight.
We experiment with two ad-hoc collections: Ro-bust04 3 and GOV2 making it more challenging to train an end-to-end ranker.
We compare PARADE against the following traditional and neural baselines: BM25 is an unsupervised ranking model based on IDF-weighted counting (Robertson et al., 1995) . The documents retrieved by BM25 also serve as the candidate documents used with reranking methods.
BM25+RM3 is a query expansion model based on RM3 (Lavrenko and Croft, 2001) . We used Anserini's (Yang et al., 2018) implementations of BM25 and BM25+RM3. Documents are indexed and retrieved with the default settings for keywords queries. For description queries, we set b = 0.6 and changed the number of expansion terms to 20.
Birch (MS) and Birch (MS→ MB) aggregate sentence-level evidence provided by BERT to rank documents . Birch (MS) is the fine-tuned BERT-Large model on the MSMARCO passage dataset while Birch (MS→MB) is further fine-tuned on TREC MicroBlog datasets. We use BM25 rather than BM25+RM3 as an initial ranking method for a fair comparison.
BERT-MaxP (MS) adopts the maximum score of passages within a document as an overall relevance score (Dai and Callan, 2019b) . However, rather than fine-tuning BERT-base on a Bing search log, we improve performance by fine-tuning on the MSMARCO passage ranking dataset.
To prepare the BERT model for the ranking task, we first fine-tune BERT on the MSMARCO passage ranking dataset (Nguyen et al., 2016 
Following prior work (Dai and Callan, 2019b; MacAvaney et al., 2019) , we use 5-fold crossvalidation. We set the reranking threshold to 100 on the test fold as trade-off between latency and effectiveness. The reported results are based on the average of all test folds. Performance is measured in terms of the P@20 and nDCG@20 ranking metrics using trec eval 5 .
The reranking effectiveness of PARADE is shown in Table 2 . It can be seen that the performance of PARADE Max and PARADE Attn is comparable, while nDCG@20 of PARADE Attn can always surpass PARADE Max . PARADE Avg underperforms other models by a large margin, which confirms that passages differ in their contributions to the overall relevance of a document. PARADE consistently outperforms the other models across both datasets, suggesting that the multi-head selfattention mechanism in the Transformer is a superior method for passage-level relevance aggregation.
Compared with other baseline models, Birch has two innate advantages: it uses the BERT-Large model with 3x more parameters than BERT-Base, 5 https://trec.nist.gov/trec_eval and it is an ensemble model that additionally considers the original ranking scores. Nevertheless, PARADE still outperform it, especially on description queries. 6 For BERT-MaxP, the reported results are better than those reported in (Dai and Callan, 2019b) with approximately a 0.02 nDCG@20 increase on Robust04 title queries. On the Robust04 collection with deeper judgments, PARADE outperforms BERT-MaxP significantly.
When applying PARADE to the more recent and efficiently trained LM model ELECTRA-Base (Clark et al., 2020) , PARADE's performance increases substantially. This model significantly improves over all baselines on nDCG@20 for the Robust04 collection and P@20 for both collections. These results illustrate that as Transformer pretraining techniques advance, PARADE is able to take advantage of improved pre-trained models.
In this section, we consider the following research questions:
• RQ1: How can BERT's efficiency be improved while maintaining its effectiveness? • RQ2: How does the number of document passages preserved influence effectiveness? • RQ3: Is it beneficial to rerank documents from a more effective initial ranking method? In particular, is reranking BM25+RM3 better than reranking BM25? Additionally, we study the effectiveness of PA-RADE on the TREC-COVID Challenge in Ap- 
While BERT-based models are effective at producing high-quality ranked lists, they are computationally expensive. However, the reranking task is sensitive to efficiency concerns, because documents must be reranked in real time after the user issues a query. In this section we consider two strategies for improving PARADE's efficiency, which also answers RQ1. Using a Smaller BERT Variant. As smaller models require fewer computations, we study the reranking effectiveness of PARADE when using pre-trained BERT models of various sizes, providing an important guidance for deploying a retrieval system. Pre-trained BERT models of various sizes were provided by (Turc et al., 2019) . From Table 3 , it can be seen that as the size of models is reduced, their effectiveness decline monotonously. The hidden layer size (#6 vs #7, #8 vs #9) plays a more critical role for performance than the number of layers (#3 vs #4, #5 vs #6). An example is the comparison between models #7 and #8. Model #8 performs better; it has fewer layers but contains more parameters. The number of parameters and inference time are also given in Table 3 to facilitate the study of tradeoffs between model complexity and effectiveness.
Distilling Knowledge from a Large Model. To further explore the limits of smaller PARADE models, we apply knowledge distillation to leverage knowledge from a large teacher model. We use PA-RADE trained with BERT-Base on the target collection as the teacher model. Smaller student models then learn from the teacher at the output level. We use mean squared error as the distilling objective, which has been shown to work effectively (Tahami et al., 2020; Tang et al., 2019) . The learning objective penalizes the student model based on both the ground-truth and the teacher model:
where L CE is the cross-entropy loss with regard to the logit of the student model and the ground truth, α weights the importance of the learning objectives, and z t and z s are logits from the teacher model and student model, respectively.
As shown in Table 3 , the nDCG@20 of distilled models always increases. The PARADE model using 8 layers (#4) can achieve comparable results with the teacher model. Moreover, the PARADE model using 10 layers (#3) can outperform the teacher model with 11% fewer parameters. The PA-RADE model trained with BERT-Small achieves a nDCG@20 above 0.5, which outperforms BERT-MaxP using BERT-Base, while requiring only 1.14 ms to perform inference on one document. The inference time for each query is only 0.114 second by reranking top 100 documents.
One hyper-parameter in PARADE is the maximum number of passages being used, i.e., preserved data size, which is studied to answer RQ2 in this section. We consider title queries on the GOV2 dataset given that these documents are longer on average than in Robust04. Figure 1 depicts nDCG@20 of PARADE with the number of passages varying from 8 to 64. Generally, larger preserved data size results in better performance for PARADE, which suggests that a document can be better un- derstood from document-level context with more preservation of its content. For PARADE Max and PARADE Attn , however, the performance degrades a little when using 64 passages. Both max-pooling (Max) and simple attention mechanism (Attn) have limited capacity and are challenged when dealing with such longer documents. PARADE is able to improve nDCG@20 as the number of passages increases, demonstrating its superiority in identifying relevant and non-relevant documents when documents become much longer. However, considering more passages also increases the number of computations performed. One advantage of the PARADE models is that the number of parameters remains constant as the number of passages in a document varies. Thus, we consider the impact of varying the number of passages considered between training and inference. As shown in Table 4 , rows indicate the number of passages considered at training time while columns indicate the number used to perform inference. The diagonal indicates that preserving more of the passages in a document consistently improves nDCG. Similarly, increasing the number of passages considered at inference time (columns) or at training time (rows) usually improves nDCG. In conclusion, the number of passages considered plays a crucial role in PARADE's effectiveness. When trading off efficiency for effectiveness, PARADE models' effectiveness can be improved by training on more passages than will be used at inference time. This generally yields a small nDCG increase.
Query expansion methods based on pseudorelevance feedback, like RM3 (Lavrenko and Croft, 2001) and NPRF (Li et al., 2018) , have been shown to increase the effectiveness of a search system. The use of PRF methods in prior work on BERT ranking models varies, however. Thus, in this section we consider the question (i.e., RQ3) of whether reranking a stronger initial ranking method (e.g., RM3) improves retrieval results. To do so, we compare the reranking effectiveness of PARADE on top of BM25 and BM25+RM3. To simplify the analysis, we focus on the ranking distribution of relevant documents. On the Robust04 dataset with title queries, we examine the top 1,000 documents retrieved by BM25 and BM25+RM3. We then divide all relevant documents retrieved into three partitions, D both , D BM 25 and D QE , defined as follows:
• D both : the relevant documents retrieved by both BM25 and BM25+RM3 • D BM 25 : the relevant documents retrieved by BM25 but not retrieved by BM25+RM3 • D QE : the relevant documents retrieved by BM25+RM3 but not retrieved by BM25 For all methods, D both is the same; differences come from D BM 25 , D QE , and non-relevant documents. In total, Count(D both ) = 9863, Count(D QE ) = 1538, and Count(D BM 25 ) = 409, which means that BM25 and BM25+RM3 share a large number of relevant documents.
Different from the previous setting, we set the reranking threshold to 1,000 to increase recall. The most effective PARADE is adopted as a reranker. The (re-)ranking effectiveness of these models is shown in Table 5 . Replacing BM25 with BM25+RM3 increases Recall@1k by about 8% and MAP@1k by about 4%, which may be a result of the nearly 1,000 relevant documents introduced by RM3. The differences for the other metrics are minor, with RM3 slightly reducing P@20. These findings are in line with recent work demonstrating that there is little difference in effec- tiveness between reranking BM25 and reranking BM25+RM3 (Nogueira et al., 2020) .
To investigate why there is little difference between reranking BM25 and BM25+RM3 for metrics considering top positions, we provide four sub-figures in Figure 2 that depict the number of relevant documents placed in different position bins (averaged by the number of queries). Figures 2a, 2b, 2c , 2d depict the ranking distribution of BM25+RM3, PARADE (reranking BM25+RM3), BM25, PARADE (reranking BM25), respectively. Due to the change in bin size from 10 to 100, there is a steep increase in the bin 101-200 across all figures. The distribution is mono-decreasing if the bin size is unchanged. It can be seen that:
• From figures 2a and 2c, the documents from D QE and D BM 25 are more likely to be ranked at the low positions (behind 100) by the initial ranking models, which suggests that both models are less confident in these documents. For BM25+RM3, it might be that the documents from D QE are mostly retrieved by the expanded terms; for BM25, it may be these documents are retrieved by terms with lower weights. • Comparing Figure 2a with 2b, as well as Figure 2c with 2d, the documents from D QE and D BM 25 can be boosted to higher positions by PARADE. Mostly, documents in D QE are retrieved using the expanded terms. PARADE can boost these documents without even knowing these terms, which confirms contextualization benefits by BERT. • Comparing Figure 2c with 2d, it can be seen that a large amount of documents from D BM 25 , especially the documents behind position 100, are boosted to higher positions, which closes the large gap in MAP between BM25 and BM25+RM3 as shown in Table 5 . The advantage of using BM25+RM3 may be that its relevance scores are good source for model ensemble. As shown in Table 5 , an ensemble method that linearly interpolates the scores achieves the best results. In conclusion, while BM25+RM3 does retrieve more relevant documents than BM25, these documents are not effectively utilized by the reranking methods. BM25+RM3 is thus more of a reranking method than an initial ranking method.
We proposed the PARADE end-to-end document reranking model and demonstrated its effectiveness on two TREC ad-hoc benchmark collections.
Our results indicate the importance of incorporating diverse relevance signals from the full text into ad-hoc ranking, rather than basing it on a single passage. We additionally investigated how model size and the initial ranking method affect performance. Knowledge distillation on PARADE boosts the performance of smaller PARADE models while substantially reducing their parameters. In response to the urgent demand for reliable and accurate retrieval of COVID-19 academic literature, TREC has been developing the TREC-COVID challenge to build a test collection during the pandemic (Voorhees et al., 2020) . The challenge uses the CORD-19 data set (Wang et al., 2020a) , which is a dynamic collection enlarged over time. There are supposed to be 5 rounds for the researchers to iterate their systems. TREC develops a set of COVID-19 related topics, including queries (keyword based), questions, and narratives. A retrieval system is supposed to generate a ranking list corresponding to these queries.
We began submitting PARADE runs to TREC-COVID from Round 2. The Round 5 results are not yet available at the time of writing. By using PARADE, we are able to utilize the full-text of the COVID-19 academic papers. We used the question topics since it works much better than other types of topics. In all rounds, we employ the full PARADE model. In Round 3, we additionally tested PARADE Attn and a combination of PARADE and PARADE Attn using reciprocal rank fusion (Cormack et al., 2009) .
Results from TREC-COVID Rounds 2-4 are shown in Table 6, Table 7, and Table 8 , respectively. 7 In Round 2, PARADE achieves the highest nDCG, further supporting its effectiveness. 8 In Round 3, our runs are not as competitive as the previous round. One possible reason is that the collection doubles from Round 2 to Round 3, which can introduce more inconsistencies between training and testing data as we trained PARADE on Round 2 data and tested on Round 3 data. In particular, our run mpiid5 run3 performed poorly. We found that it tends to retrieve more documents that are not likely to be included in the judgment pool. When considering the bpref metric that takes only the judged documents into account, its performance is comparable to that of the other variants. As measured by nDCG, PARADE's performance improved in Round 4 (Table 8 ), but is again outperformed by other approaches. It is worth noting that the PARADE runs were created by single models (excluding the fusion run from Round 3), whereas e.g. the UPrrf38rrf3-r4 run in Round 4 is an ensemble of more than 20 runs.
Ranking Task
The MSMARCO document ranking dataset 9 is a large-scale collection and is used in TREC 2019 Deep Learning track (Craswell et al., 2019 MaxP (Dai and Callan, 2019b) as the reranking method, TUW19-d3-re (Hofstätter et al., 2019) is a Transformer-based non-BERT method, and idst bert r1 (Yan et al., 2019) utilizes struct-BERT (Wang et al., 2020b) , which is intended to strengthen the modeling of sentence relationships. All PARADE variants outperform ucas runid1 and TUW19-d3-re in terms of nDCG@10, but cannot outperform idst bert r1. Since this run's pre-trained structBERT model is not publicly available, we are not able to embed it into PARADE and make a fair comparison. In contrast with the previous results, the other variants outperform PARADE in this setting.
As previously described, the BERT model used in PARADE is fine-tuned on the MSMARCO passage ranking dataset before being embedded into PA-RADE. This training set consists of approximately 400M tuples of query, relevant passage, and nonrelevant passage. The dev set and test set consist of approximately 6,900 and 6,800 queries, respectively. For each passage, we use BERT's [CLS] vector as in Equation 1 to a single-layer feed-forward network to obtain the probability of being relevant. We follow the training setup in and fine-tune the model with a batch size of 32 for 400k iterations. After that, the fine-tuned model is used as weight initialization in the BERT layers of PARADE.
As mentioned in , finetuning BERT on different domains can result in different model effectiveness. We verify the performance of PARADE using the BERT models fine-tuned on the above mentioned MSMARCO domain as well as Bing search log 10 . Results on Robust04 when using the original BERT model, a BERT model fine-tuned on Bing search logs, and using a BERT model fine-tuned on MSMARCO are shown in Table 10 . It can be seen that fine-tuning on MSMARCO outperforms the other approaches by a large margin. Considering the model size, BERT-Base shows comparable ability to BERT-Large while requiring fewer parameters.
In our pilot study, we tune learning rates from {1e-6, 3e-6, 5e-6, 1e-5, 5e-5}, α for knowledge distillation from {0.25, 0.5, 0.75}, numbers of Transformer layers from 1 to 4, and the numbers of training epochs from 1 to 10. Then we fix the learning rate as 3e-6, the number of Transformer layers as 2, the number of training epochs as 3, and α as 0.75 for all experiments. For PARADE, the configuration of Transformer layers (e.g., number of attention heads, hidden size, etc.) is the same as the Transformer block being used in BERT.
Documents are split into passages. We set the maximum number of passages in each document as 16 and 32 for Robust04 and GOV2 respectively. As we split the documents using a sliding window of 150 words with a stride of 100 words, a maximum number of 1650 words in each document are retained on the Robust04 collection while 3250 on GOV2. The maximum sequence length in BERT is set as 256. When running PARADE, documents with less number of required passages are padded and later masked out by passage level masks. For documents longer than required, the first and last passages are always kept while the remaining are selected using a uniform sampling strategy as in (Dai and Callan, 2019b) .