Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection (2022)

Abstract: An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents. In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents, to improve the performance of transformers for AS2, and mitigate the requirement of large labeled datasets. Specifically, the model is tasked to predict whether: (i) two sentences are extracted from the same paragraph, (ii) a given sentence is extracted from a given paragraph, and (iii) two paragraphs are extracted from the same document. Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models such as RoBERTa and ELECTRA for AS2.

## introduction question answering (qa) finds itself at the core of several commercial applications, for e.g., virtual assistants such as google home, alexa and siri. answer sentence selection (as2) is an important task for qa systems operating on unstructured text such as web documents. when presented with a set of relevant documents for a question (retrieved from a web index), as2 aims to find the best answer sentence for the question. the recent popularity of pre-trained transformers @xcite @xcite , has made them the de-facto approach for most qa tasks, including as2. several research works @xcite @xcite fine-tune transformers for as2, by posing it as a sentence-pair task and performing inference over the encoded representations of the question and answer candidates. as2 is a knowledge-intensive complex reasoning task, where the answer candidates for a question can stem from multiple documents, possibly on different topics linked to concepts in the question. while there have been recent works @xcite proposing pretraining strategies for obtaining multi-document aware document representations over long input encoders such as the longformer @xcite , there has been limited research @xcite on enhancing sentence-pair representations with paragraph and document level semantics. furthermore, obtaining high quality human labeled examples for as2 is expensive and time consuming, due to the large number of answer candidates to be annotated for each question. domainspecific as2 datasets such as wikiqa @xcite and trec-qa @xcite only contain a few thousand questions. @xcite show that effectively fine-tuning pre-trained transformers on these domain specific as2 datasets requires an intermediate fine-tuning transfer on a large scale as2 dataset (asnq). towards improving the downstream performance of pre-trained transformers for as2 and mitigating the requirement of large scale labeled data for fine-tuning, we propose three novel sentencelevel transformer pre-training objectives, which can incorporate paragraph-level semantics across multiple documents. analogous to the sentence-pair nature of as2, we design our pre-training objectives to operate over a pair of input text sequences. the model is tasked with predicting: (i) whether the sequences are two sentences extracted from the same paragraph, (ii) whether the first sequence is a sentence that is extracted from the second sequence (paragraph), and (iii) whether the sequences are two paragraphs belonging to the same document. we evaluate our paragraph-aware pre-trained transformers for as2 on three popular public datasets: asnq, wikiqa and trec-qa; and one industrial qa benchmark 1 . results show that our pre-training can improve the performance of finetuning baseline transformers such as roberta and electra on as2 by ∼3-4% points without requiring any additional data (labeled/unlabeled). ## related work answer sentence selection (as2) earlier approaches for as2 used cnns @xcite or alignment networks @xcite to learn and score question and answer representations. since then, compareand-aggregate architectures have also been extensively studied @xcite @xcite . garg et al. achieved state-of-the-art results by fine-tuning transformers on a large qa corpora first, and then adapting to a smaller as2 dataset. token-level pre-training objectives masked language modeling (mlm) is one of the most popular token-level pre-training objectives used for transformers @xcite . some other models trained using token-level pretraining objectives are @xcite and @xcite . @xcite modify mlm to a span-prediction objective to make the model generalize well to machine reading tasks in qa. sentence-level pre-training objectives in addition to mlm, @xcite uses the next sentence prediction (nsp) objective, which was later shown to not provide empirically improvements over mlm by @xcite (possibly due to the task being very simple). @xcite propose a sentence order prediction (sop) objective. @xcite enhance nsp to a multiplechoice prediction of the next sentence over a set of candidates, however they embed each sentence independently without cross-attention between them similar to @xcite . @xcite propose a supervised contrastive learning approach for enhancing sentence representations for textual similarity tasks. paragraph/document-level semantics @xcite pre-train bi-hlstms for obtaining hierarchical document representations. hib-ert @xcite uses document-level token masking and sentence masking pre-training objectives for generative tasks such as document summarization. transformer pre-training objec-1 we will release code and pre-trained models at https: //github.com/amazon-research/wqa-pretraining tives at different granularities of document semantics are discussed in @xcite for fact verification, and in @xcite for retrieval. ginzburg et al.; @xcite propose pretraining strategies for document embeddings for retrieval tasks such as document-matching. de-clutr @xcite uses contrastive learning for cross-encoding two sentences coming from the same/different documents in a transformer, and is evaluated on pairwise binary classification tasks like natural language inference. our work differs from this since we use a cross-encoder architecture to capture cross-attention between the question and answer, and evaluate our approach on the relevance ranking task of as2 over hundreds of candidates. contemporary works @xcite pretrain transformers using paragraph-aware objectives for multi-sentence inference tasks. our work differs from this since we only encode a pair of sentences using the transformer, while the former encode multiple sentences and use sophisticated prediction heads to aggregate information across multiple representations. transformers for long inputs longformer (beltagy et al., 2020), big bird (zaheer et al., 2020), etc. model very long inputs (e.g, entire documents) by reducing the complexity of transformer attention. this provides longer context, which is useful for machine reading and summarization. ## answer sentence selection (as2) in this section we formally define the task of as2. given a question q and a set of answer candidates @xmath0, . . ., a n }, the objective is to select the candidate ā ∈ a that best answers q. as2 can be modeled as a ranking task over a to learn a scoring function f : q×@xmath1, a) of an answer candidate a being correct. the best answer ā corresponds to argmax n @xmath2, a i ). pre-trained transformers are used as qa pair encoders for as2 to approximate the function f . spans in same paragraph (ssp) given two sequences (a, b) as input to the transformer, the objective is to predict if a and b belong to the same paragraph in a document. to create positive pairs (a, b), given a document d, we extract two small, contiguous and disjoint subsets of sentences to be used as a and b from a single paragraph p @xmath3. to create negative pairs, we sample spans of sentences b ′ from different paragraphs p j , j ̸ = i in the same document d (hard negatives) and also from different documents (easy negatives). the negative pairs correspond to (a, b ′ ). posing the above pre-training objective in terms of spans (instead of sentences) allows for modifying the lengths of the inputs a, b (by changing number of @xmath4, b). when fine-tuning transformers for as2, typically the question is provided as the first input and a longer answer candidate/paragraph is provided as the second input. for our experiments (sec 5), we use a longer span for input b than a. span in paragraph (sp) given two sequences (a, b) as input to the transformer, the objective is to predict if a is a span of text extracted from a paragraph b in a document. to create positive pairs (a, b), given a paragraph p i in a document d, we extract a small contiguous span of sentences a from it and create the input pair as (a, p i \ a). to create negative pairs, we select other paragraphs p j , j ̸ = i in the same document d and remove a randomly chosen span a ′ from each of them. the negative pairs correspond to (a, p j \ a ′ ). this is necessary to ensure that the model does not simply recognize whether the second input is a complete paragraph or a clipped version. to create easy negatives, we use the above approach for paragraphs p j sampled from documents other than d. paragraphs in same document (psd) given two sequences (a, b) as input to the transformer, the objective is to predict if a and b are paragraphs belonging to the same document. to create positive pairs (a, b), given a document d k , we randomly select paragraphs p i , p @xmath5, p j ). to create negative pairs, we randomly select p ′ j / ∈ d k , and obtain a pair (p i , p ′ j ). ## conclusion in this paper we have presented three sentencelevel pre-training objectives for transformers to incorporate paragraph and document-level semantics. our objectives predict whether (i) two sequences are sentences extracted from the same paragraph, (ii) first sequence is a sentence extracted from the second, and (iii) two sequences are paragraphs belonging to the same document. we evaluate our pre-trained models for the task of as2 on four datasets. our results show that our pre-trained models outperform the baseline transformers such as roberta and electra.