Paragraph-based Transformer Pre-training for Multi-Sentence Inference
Abstract
Inference tasks such as answer sentence selection (AS2) or fact veriﬁcation are typically solved by ﬁne-tuning transformer-based models as individual sentence-pair classiﬁers.
Recent studies show that these tasks beneﬁt from modeling dependencies across multiple candidate sentences jointly.
In this paper, we ﬁrst show that popular pre-trained transformers perform poorly when used for ﬁne-tuning on multi-candidate inference tasks.
We then propose a new pre-training objective that models the paragraph-level semantics across multiple input sentences.
Our evaluation on three AS and one fact veriﬁcation datasets demonstrates the superiority of our pre-training technique over the traditional ones for transformers used as joint models for multi-candidate inference tasks, as well as when used as cross-encoders for sentence-pair formulations of these tasks.
Introduction
Pre-trained transformers (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020) have become the de facto standard for several NLP applications, by means of ﬁne-tuning on downstream data.
The most popular architecture uses self-attention mechanisms for modeling long range dependencies between compounds in the text, to produce deep contextualized representations of the input.
There are several downstream NLP applications that require reasoning across multiple inputs candidates jointly towards prediction.
Some popular examples include (i) Answer Sentence Selection (AS2) (Garg et al., 2020), which is a Question Answering (QA) task that requires selecting the best answer from a set of candidates for a question; and (ii) Fact Veriﬁcation (Thorne et al., 2018), which reasons whether a claim is supported/refuted by multiple evidences.
Inherently, these tasks can utilize information from multiple candidates (answers/evidences) to support the prediction of a particular candidate.
Pre-trained transformers such as BERT are used for these tasks as cross-encoders by setting them as sentence-pair classiﬁcation problems, i.e, aggregating inferences independently over each candidate.
Recent studies (Zhang et al., 2021; Tymoshenko and Moschitti, 2021) have shown that these tasks beneﬁt from encoding multiple candidates together, e.g., encoding ﬁve answer candidates per question in the transformer, so that the cross-attention can model dependencies between them.
However, Zhang et al.
only improved over the pairwise cross-encoder by aggregating multiple pairwise cross-encoders together (one for each candidate), and not by jointly encoding all candidates together in a single model.
In this paper, we ﬁrst show that popular pretrained transformers such as RoBERTa perform poorly when used for jointly modeling inference tasks (e.g., AS2) using multi-candidates.
We show that this is due to a shortcoming of their pre-training objectives, being unable to capture meaningful dependencies among multiple candidates for the ﬁnetuning task.
To improve this aspect, we propose a new pre-training objective for ‘joint’ transformer models, which captures paragraph-level semantics across multiple input sentences.
Speciﬁcally, given a target sentence s and multiple sentences (from the same/different paragraph/document), the model needs to recognize which sentences belong to the same paragraph as s in the document used.
Joint inference over multiple-candidates entails modeling interrelated information between multiple short sentences, possibly from different paragraphs or documents.
This differs from related works (Beltagy et al., 2020; Zaheer et al., 2020; Xiao et al., 2021) that reduce the asymptotic complexity of transformer attention to model long contiguous inputs (documents) to get longer context for tasks such as machine reading and summarization.
We evaluate our pre-trained multiple-candidate based joint models by (i) performing AS2 on
ASNQ (Garg et al., 2020), WikiQA (Yang et al., 2015), TREC-QA (Wang et al., 2007) datasets; and (ii) Fact Veriﬁcation on the FEVER (Thorne et al., 2018) dataset.
We show that our pre-trained joint models substantially improve over the performance of transformers such as RoBERTa being used as joint models for multi-candidate inference tasks, as well as when being used as cross-encoders for sentence-pair formulations of these tasks.
Related Work
Multi-Sentence Inference: Inference over a set of multiple candidates has been studied in the past (Bian et al., 2017;
Ai et al., 2018).
The most relevant for AS2 are the works of Bonadiman and
Moschitti (2020) and Zhang et al. (2021), the former improving over older neural networks but failing to beat the performance of transformers; the latter using task-speciﬁc models (answer support classiﬁers) on top of the transformer for performance improvements. For fact veriﬁcation, Tymoshenko and Moschitti (2021) propose jointly embedding multiple evidence with the claim towards improving the performance of baseline pairwise cross-encoder transformers.
Transformer pre-training Objectives: Masked
Language Modeling (MLM) is a popular transformer pre-training objective (Devlin et al., 2019;
Liu et al., 2019).
Other models are trained using token-level (Clark et al., 2020; Joshi et al., 2020; Yang et al., 2019;
Liello et al., 2021) and/or sentence-level (Devlin et al., 2019; Lan et al., 2020;
Wang et al., 2020) objectives.
REALM (Guu et al., 2020) uses a differentiable neural retriever over
Wikipedia to improve MLM pre-training.
This differs from our pre-training setting as it uses additional knowledge to improve the pre-trained LM.
DeCLUTR (Giorgi et al., 2021) uses a contrastive learning objective for cross-encoding two sentences coming from the same/different documents in a transformer.
DeCLUTR is evaluated for sentencepair classiﬁcation tasks and embeds the two inputs independently without any cross-attention, which differs from our setting of embedding multiple candidates jointly for inference.
Modeling Longer Sequences:
Beltagy et al. (2020); Zaheer et al.
(2020) reduce the asymptotic complexity of transformer attention to model very long inputs for longer context.
For tasks with short sequence lengths, LongFormer works on par or slightly worse than RoBERTa (attributed to re- duced attention computation).
These works encode a single contiguous long piece of text, which differs from our setting of having multiple short candidates, for a topic/query, possibly from different paragraphs and documents.
DCS (Ginzburg et al., 2021) proposes a cross-encoder for the task of document-pair matching. DCS is related to our work as it uses a contrastive pre-training objective over two sentences extracted from the same paragraph, however different from our joint encoding of multiple sentences, DCS individually encodes the two sentences and then uses the InfoNCE loss over the embeddings. CDLM (Caciularu et al., 2021) specializes the Longformer for documentpair matching and cross-document coreference resolution. While the pre-training objective in CDLM exploits information from multiple documents, it differs from our setting of joint inference over multiple short sentences.
Multi-Sentence Transformers Models 3. Multi-sentence Inference Tasks
AS2: We denote the question by q , and the set of answer candidates by C = { c 1 , . . .
c n } .
The objective is to re-rank C and ﬁnd the best answer A for q. AS2 is typically treated as a binary classiﬁcation task: ﬁrst, a model f is trained to predict the correctness/incorrectness of each c i ; then, the candidate with the highest likelihood of being correct is selected as an answer, i.e. , A = argmax ni =
1 f ( c i ) .
Intuitively, modeling interrelated information between multiple c i ’s can help in selecting the best answer candidate (Zhang et al., 2021).
Fact Veriﬁcation: We denote the claim by F , and the set of evidences by C
= { c 1 . . .
c n } that are retrieved using DocIR.
The objective is to predict whether F is supported/refuted/
neither using C (at least one evidence c i is required for supporting/refuting F ).
Tymoshenko and Moschitti (2021) jointly model evidences for supporting/refuting a claim as they can complement each other.
3. Joint Encoder Architecture
For jointly modeling multi-sentence inference tasks, we use a monolithic transformer crossencoder to encode multiple sentences using selfattention as shown in Fig 1.
To perform joint inference over k sentences for question q or claim F , the model receives concatenated sentences [ s 0 . . .
s k ] as input, where the ﬁrst sentence is either the question or the claim ( s 0 =
q
or s 0 = F ), and the remainder are k candidates s i
=
c
i , i = { 1 . . .
k } .
We pad (or truncate) each sentence s i to the same ﬁxed length L (total input length L × ( k + 1) ), and use the embedding for the [CLS] /
[SEP] token in front of each sentence s i as its embedding (denoted by E i ).
Similar to Devlin et al.
, we create positional embeddings of tokens using integers 0 to L ( k +1) − 1 , and extend the token type ids from { 0 , 1 } to { 0 . . .
k } corresponding to ( k + 1) input sentences.
3. Inference using Joint Transformer Model
We use the output embeddings [ E 0 . . .
E k ] of sentences for performing prediction as following:
Predicting a single label: We use two separate classiﬁcation heads to predict a single label for the input to the joint model [ s 0 . . .
s k ] : (i) IE 1 : a linear layer on the output embedding E 0 of s 0 (similar to
BERT) referred to as the Individual Evidence (IE 1 ) inference head, and (ii) AE 1 : a linear layer on the average of the output embeddings [ E 0 , E 1 , . . .
, E k ] to explicitly factor in information from all candidates, referred to as the Aggregated Evidence (AE 1 ) inference head.
For Fact Veriﬁcation, we use prediction heads IE 1 and AE 1 .
Predicting Multiple Labels: We use two separate classiﬁcation heads to predict k labels, one label each for every input [ s 1 . . .
s k ] speciﬁc to s 0 :
(i)
IE k : a shared linear layer applied to the output embedding E i of each candidate s i , i ∈ { 1 . . .
k } referred to as k -candidate Individual Evidence (IE k ) inference head, and (ii) AE k : a shared linear layer applied to the concatenation of output embedding
E 0 of input s 0 and the output embedding E i of each candidate s i , i ∈ { 1 . . .
k } referred to as kcandidate Aggregated Evidence (AE k ) inference head.
For AS2, we use prediction heads IE k and
AE k .
Prediction heads are illustrated in Figure 2.
3. Pre-training with Paragraph-level Signals
Long documents are typically organized into paragraphs to address the document’s general topic from different viewpoints.
The majority of trans- former pre-training strategies have not exploited this rich source of information, which can possibly provide some weak supervision to the otherwise unsupervised pre-training phase.
To enable joint transformer models to effectively capture dependencies across multiple sentences, we design a new pre-training task where the model is (i) provided with ( k + 1) sentences { s 0 . . .
s k } , and (ii) tasked to predict which sentences from { s 1 . . .
s k } belong to the same paragraph P as s 0 in the document D .
We call this pre-training task Multi-Sentences in
Paragraph Prediction ( MSPP ).
We use the IE k and
AE k prediction heads, deﬁned above, on top of the joint model to make k predictions p i corresponding to whether each sentence s i , i ∈{ 1 . . .
k } lies in the same paragraph P ∈ D as s 0 .
More formally: p i = (cid:40) 1 if s 0 , s i ∈ P in D [-
-]0 otherwise ∀ i = { 1 , . . .
, k }
We randomly sample a sentence from a paragraph
P in a document D to be used as s 0 , and then (i) randomly sample k 1 sentences (other than s 0 ) from P as positives, (ii) randomly sample k 2 sentences from paragraphs other than P in the same document D as hard negatives, and (iii) randomly sample k 3 sentences from documents other than D as easy negatives (note that k 1 + k 2 + k 3 = k ).
Experiments
We evaluate our joint transformers on three AS and one Fact Veriﬁcation datasets 1 .
Common LM benchmarks, such as GLUE (Wang et al., 2018), are not suitable for our study as they only involve sentence pair classiﬁcation.
4. Datasets Pre-training:
To eliminate any improvements stemming from usage of more data, we perform pre-training on the same corpora as RoBERTa: English Wikipedia, the BookCorpus, OpenWebText and CC-News.
For our proposed pre-training, we randomly sample sentences from paragraphs as s 0 , [Table] and choose k 1 =1 , k 2 =2 , k 3 =2 as the speciﬁc values for creating positive and negative candidates for s 0 .
For complete details refer to Appendix A.
Fine-tuning: For AS2, we compare performance with MAP, MRR and Precision of top ranked answer (P@1).
For fact veriﬁcation, we measure
Label Accuracy (LA).
Brief description of datasets is presented below (details in Appendix A):
• ASNQ:
A large AS2 dataset (Garg et al., 2020) derived from NQ (Kwiatkowski et al., 2019), where the candidate answers are from Wikipedia pages and the questions are from search queries of the Google search engine. We use the dev.
and test splits released by Soldaini and Moschitti. • WikiQA: An AS2 dataset (Yang et al., 2015) where the questions are derived from query logs of the Bing search engine, and the answer candidate are extracted from Wikipedia. We use the most popular clean setting (questions having at least one positive and one negative answer).
• TREC-QA: A popular AS2 dataset (Wang et al., 2007)
containing factoid questions.
We only retain questions with at least one positive and one negative answer in the development and test sets.
• FEVER:
A dataset for fact extraction and veriﬁcation (Thorne et al., 2018) to retrieve evidences given a claim and identify if the evidences support/refute the claim. As we are interested in the fact veriﬁcation sub-task, we use evidences retrieved by Liu et al. using a BERT-based DocIR.
4. Experimental Details and Baselines
We use k =5 for our experiments (following (Zhang et al., 2021) and (Tymoshenko and Moschitti, 2021)), and perform continued pre-training starting from RoBERTa-Base using a combination of MLM and our MSPP pre-training for 100k steps with a batch size of 4,096.
We use two different prediction heads, IE k and AE k , for pre-training.
For evaluation, we ﬁne-tune all models on the downstream AS2 and FEVER datasets using the corre- [Table] Statistically signiﬁcant results (Student t -test 95% ) are underlined. Complete results in Appendix C. sponding IE k and AE k prediction heads.
We consider the pairwise RoBERTa-Base cross-encoder and RoBERTa-Base LM used as a joint model with
IE k and AE k prediction heads as the baseline for AS2 tasks.
For FEVER, we use several baselines:
GEAR (Zhou et al., 2019), KGAT (Liu et al., 2020),
Transformer-XH (Zhao et al., 2020), and three models from (Tymoshenko and Moschitti, 2021): (i) Joint RoBERTa-Base with IE 1 prediction head, (ii) Pairwise RoBERTa-Base with max-pooling, and (iii) weighted-sum heads.
For complete experimental details, refer to Appendix B.
4. Results
Answer Sentence Selection:
The results for AS tasks are presented in Table 1, averaged across ﬁve independent runs.
From the table, we can see that the RoBERTa-Base when used as a joint model for multi-candidate inference using either the IE k or AE k prediction heads performs inferior to RoBERTa-Base used as a pairwise cross-encoder.
Across ﬁve experimental runs, we observe that ﬁnetuning RoBERTa-Base as a joint model faces convergence issues (across various hyper-parameters) indicating that the MLM pre-training task is not sufﬁcient to learn text semantics which can be exploited for multi-sentence inference.
Our MSPP pre-trained joint models (with both
IE k , AE k heads) get signiﬁcant improvements over the pairwise cross-encoder baseline and very large improvements over the RoBERTa-Base joint model.
The former highlights modeling improvements stemming from joint inference over multiple- [Table] candidates, while the latter highlights improvements stemming from our MSPP pre-training strategy.
Across all three AS2 datasets, our joint models are able to get the highest P@1 scores while also improving the MAP and MRR metrics.
To demonstrate that our joint models can effectively use information from multiple candidates towards prediction, we perform a study in Tablewhere the joint models are used to re-rank the top- k candidates ranked by the pairwise RoBERTa-Base cross-encoder.
Our joint models can signiﬁcantly improve the P@1 over the baseline for all datasets.
The performance gap stems from questions for which the pairwise RoBERTa model was unable to rank the correct answer at the top position, but support from other candidates in the top-k helped the joint model rank it in the top position.
Fact Veriﬁcation: The results for the FEVER task are presented in Table 3 and show that our joint models (pre-trained with both the IE k and AE k heads and ﬁne-tuned with the IE 1 and AE 1 heads) outperform all previous baselines considered, including the RoBERTa-Base joint model directly applied for multi-sentence inference.
Compute Overhead: We present a simpliﬁed latency analysis for AS2 (assuming sentence length L ) as follows: a pairwise cross-encoder uses k transformer steps with input length 2 L , while our model uses 1 step with input length ( k +1)
× L .
Since transformer attention scales quadratic on input length, our model should take ( k +1) 2 4 k times the inference time of the cross-encoder, which is 1 .when k =5 .
However, when we ﬁne-tune for WikiQA on one A100-GPU, we only observe latency increasing from 71 s → 81 s (only 14 . 1% increase).
The input embeddings and feedforward layers vary
[Table]
inFebruary1862intheMississippiRiverandwaslatertowedtoLakePontchartrainforadditionaltrials
[Table] takenintogovernmentserviceunderthecontroloftheConfederateStatesArmyatCharleston [Table] linearly with input length, reducing overheads of self-attention.
Refer to Appendix C.3 for details.
Qualitative Examples: We present some qualitative examples from the three AS2 datasets highlighting cases where the pairwise RoBERTa-Base model is unable to rank the correct answer on the top position, but our pre-trained joint model (Joint
MSPP IE k → FT IE k ) can do this using supporting information from other candidates in Table 4.
Conclusions
In this paper we have presented a multi-sentence cross-encoder for performing inference jointly on multiple sentences for tasks like answer sentence selection and fact veriﬁcation.
We have proposed a novel pre-training task to capture paragraph-level semantics.
Our experiments on three answer selection and one fact veriﬁcation datasets show that our pre-trained joint models can outperform pairwise cross-encoders and pre-trained LMs when directly used as joint models.
Appendix A Datasets
We present the complete details for all the datasets used in this paper along with links to download them for reproducibility of results.
A. Pre-training Datasets
We use the Wikipedia 2 , BookCorpus 3 , OpenWebText (Gokaslan and Cohen, 2019) and CC-Newsdatasets for performing pre-training of our joint transformer models.
We do not use the STORIES dataset as it is no longer available for research use 5 .
After decompression and cleaning we obtained 6GB, 11GB, 38GB and 394GB of raw text respectively from the BookCorpus, Wikipedia, OpenWebText and CC-News.
A. Finetuning Datasets
We evaluate our joint transformers on three AS and one Fact Veriﬁcation datasets.
The latter differs from the former in not selecting the best candidate, but rather explicitly using all candidates to predict the target label.
Here are the details of the ﬁnetuning datasets that we use for our experiments along with data statistics for each dataset:
Dataset Split # Questions # Candidates Avg.
# C/Q A S NQ
Train 57, 20,377, 356.
Dev 1, 463, 347.
[Table]
Test 1, 466, 348.
W i k
i
QA
Train 2, 20,
[Table]
Dev
[Table]
[Table]
Test 2, 9. T R E CQA Train 1, 53, [Table]
[Table]
Dev 1, 19.
[Table]
Test 1, 21.
• ASNQ:
A large-scale AS2 dataset (Garg et al., 2020) 6 where the candidate answers are from Wikipedia pages and the questions are from search queries of the Google search engine.
ASNQ is a modiﬁed version of the Natural Questions
(NQ) (Kwiatkowski et al., 2019) dataset by converting it from a machine reading to an AS2 dataset.
This is done by labelling sentences from the long answers which contain the short answer string as positive correct answer candidates and all other answer candidates as negatives.
We use the dev.
and test splits released by Soldaini and Moschitti 7 . • WikiQA: An AS2 dataset released by Yang et al.
8 where the questions are derived from query logs of the Bing search engine, and the answer candidate are extracted from Wikipedia.
This dataset has a subset of questions having no correct answers (all-) or having only correct answers (all+).
We remove both the all- and all+ questions for our experiments (“clean" setting).
• TREC-QA: A popular AS2 dataset released by Wang et al..
For our experiments, we trained on the train-all split, which contains more noise but also more question-answer pairs.
Regarding the dev.
and test sets we removed the questions without answers, or those having only correct or only incorrect answer sentence candidates.
This setting refers to the “clean" setting (Shen et al., 2017), which is a TREC-QA standard.
• FEVER:
A popular benchmark for fact extraction and veriﬁcation released by Thorne et al.
The aim is to retrieve evidences given a claim, and then identify whether the retrieved evidences support or refute the claim or if there is not enough information to make a choice.
For
supporting/refuting a claim, at least one of the retrieved evidences must support/retrieve the claim.
Note that the performance on FEVER depends crucially on the retrieval system and the candidates retrieved.
For our experiments, we are interested only in the fact veriﬁcation sub-task and thus we exploit the evidences retrieved by Liu et al. using a BERT-based DocIR 9 .
[Table]
[Table]
[Table]
[Table]
[Table]
B Experimental Setup B. Complete Experimental Details
Following standard practice, the token ids, positional ids and token type ids are embedded using separate embedding layers, and their sum is fed as the input to the transformer layers.
We use k = for our experiments (following Zhang et al.
; Tymoshenko and Moschitti), and perform continuous pre-training starting from the RoBERTa-Base checkpoint using a combination of MLM and
our
MSPP pre-training objective for 100,000 steps with a batch size of 4096.
We use a triangular learning rate with 10,000 warmup steps and a peak value of 5 ∗ 10 − 5 .
We use Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
999 and (cid:15) =
10 − 8 .
We apply a weight decay of 0 .
01 and gradient clipping when values are higher than 1 . 0 .
We set the dropout ratio to 0 .
and we use two different prediction heads for pre-training: IE k and AE k .
We follow the strategy of (Devlin et al., 2019; Lan et al., 2020), and equally weight the the two pre-training loss objectives: MLM and MSPP.
For evaluation, we ﬁne-tune all models on the downstream AS2 and FEVER datasets: using the same IE k and AE k prediction heads exploited in pre-training for AS2 and using either IE 1 or AEprediction heads for FEVER.
We ﬁnetune every model with the same maximum sequence length equal to 64 ∗ ( k + 1) =
384 tokens.
For ASNQ we train for up to 6 epochs with a batch size ofand a learning rate of 10 − 5 with the same Adam optimizer described above but warming up for only 5000 steps.
We do early stopping on the MAP of the development set.
For WikiQA and TREC-QA, we created batches of 32 examples and we used a learning equal to 2 ∗ 10 − 6 and 1000 warm up steps.
We train for up to 40 epochs again with early stopping on the MAP of the development set.
On
FEVER, we use a batch size of 64, a learning rate of 10 − 5 , 1000 warm up steps and we do early stopping checking the Accuracy over the development set.
We implemented our code based on HuggingFace’s
Transformers library (Wolf et al., 2020).
B. Baselines
For AS2, we consider two baselines: (i) pairwise RoBERTa-Base model when used as a crossencoder for AS2, and (ii) RoBERTa-Base LM when used as a joint model with IE k and AE k prediction heads independently for AS2 tasks.
For FEVER, we use several recent baselines from Tymoshenko and Moschitti: (i) GEAR (Zhou et al., 2019), (ii) KGAT (Liu et al., 2020), (iii)
Transformer-XH (Zhao et al., 2020), (iv) joint
RoBERTa-Base with IE 1 prediction head (Tymoshenko and Moschitti, 2021), (v) pairwise
RoBERTa-Base when used as a cross-encoder with max-pooling head (Tymoshenko and Moschitti, 2021), (vi) pairwise RoBERTa-Base when used as a cross-encoder with weighted-sum head (Tymoshenko and Moschitti, 2021).
We used metrics from Torchmetrics (Detlefsen et al., 2022) to compute MAP, MRR, Precision@ and Accuracy.
B. Metrics
The performance of AS2 systems in practical applications is typically (Garg and Moschitti, 2021) measured using the Accuracy in providing correct answers for the questions (the percentage of correct responses provided by the system), also called the
Precision-at-1 (P@1).
In addition to P@1, we use
Mean Average Precision (MAP) and Mean Reciprocal Recall (MRR) to evaluate the ranking produced of the set of candidates by the model.
For FEVER, we measure the performance using
Label Accuracy (LA), a standard metric for this dataset, that measures the accuracy of predicting support/refute/neither for a claim using a set of evidences.
C Complete Results and Discussion C. Results on AS2 with cascaded pairwise and Joint re-ranker
Below we present results of evaluating our joint models to re-rank the top- k candidates ranked by the pairwise RoBERTa-Base cross-encoder.
Our joint models can signiﬁcantly improve the P@1,
MAP and MRR over the baseline for all datasets.
The performance gap stems from questions for which the pairwise RoBERTa model was unable to rank the correct answer at the top position, but support from other candidates in the top-k helped the joint model rank it in the top position.
C. Results on FEVER
Here we present complete results on the FEVER dataset in Table 8, by also presenting some additional baselines such as: (i) pairwise BERT-Base cross-encoder (Tymoshenko and Moschitti, 2021), (ii) joint BERT-Base cross-encoder with IE 1 prediction head, (iii) DOMLIN++ (Stammbach and [Table]
[Table] Ash, 2020) which uses additional DocIR components and data (MNLI (Williams et al., 2018)) for ﬁne-tuning, (iv) DREAM (Zhong et al., 2020) that uses the XL-Net model.
Note that comparing our joint models with (iii) and (iv) is unfair since they use additional retrieval components, datasets and larger models.
We just include these results here for the sake for completeness.
Interestingly, our joint models outperform DREAM and DOMLIN++ on the dev set without using additional retrieval and larger models.
C. Compute Overhead of Joint Models
Change in Number of Model Parameters: The transformer block of our joint inference model is identical to pre-trained models such as RoBERTa, and contains the exact same number of parameters.
Classiﬁcation heads IE 1 , IE k and AE 1 all operate on the embedding of a single token, and are identical to the classiﬁcation head of RoBERTa ( AE k operates on the concatenation of two token embeddings, and contains double the number of parameters as the RoBERTa).
The maximum sequence length allowed for both the models is the same (512).
The exact number of parameters of our joint model with AE k and the RoBERTa model are 124 , 062 , 720 and 124 , 055 , 040 respectively.
Change in Inference Latency: While our joint model provides a longer input sequence to the transformer, it also reduces the number of forward passes that need to be done by a pairwise crossencoder.
A simpliﬁed latency analysis for AS (assuming each sentence has a length L ): pairwise cross-encoder will need to make k forward passes of the transformer with a sequence of length 2 L ( q with each candidate c i ), while our joint model will only need to make 1 forward pass of the transformer with input length ( k +1)
× L ( q with k candidates).
Transformer self-attention is quadratic in input sequence length, so this should lead to the inference time of out joint model being ( k +1) 2 4 k times the inference time of the cross-encoder.
However, the input embedding layer and the feedforward layers are linear in input sequence length, so this should lead to a reduction in the inference time of our joint model by ( k +1)
k times the inference time of the cross-encoder.
Empirically, when we ﬁne-tune for WikiQA on one A100-GPU, we only observe latency increasing from 71 s → 81 s (increase of only 14 . 1% ).
[-Launched : July 1863 Acquired : August 
-]