Context-Aware Transformer Pre-Training for Answer Sentence Selection (2023)

Abstract: Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.

## introduction answer sentence selection (as2) is a fundamental task in qa, which consists of re-ranking a set of answer sentence candidates according to how correctly they answer a given question. from a practical standpoint, as2-based qa systems can operate under much lower latency constraints than corresponding machine reading (mr) based qa systems. nowadays, latency is of particular importance because sources of information such as knowledge bases or web indexes may contain million or billion of documents. in as2, latency can be minimized because systems process several sentences/documents in parallel, while mr systems parse the entire document/passage in a sliding window fashion before finding the answer span @xcite . modern as2 systems @xcite ) use transformers to cross-encode question and answer candidates together. recently, @xcite proved that performing answer ranking using only the candidate sentence * work done as an intern at amazon alexa ai is sub-optimal, for e.g., the answer sentence may contain unresolved coreference with entities, or the sentence may lack specific context for answering the question. several works @xcite @xcite have explored performing as2 using context around answer candidates (for example, adjacent sentences) towards improving performance. local contextual information, i.e., the previous and next sentences of the answer candidates, can help coreference disambiguation, and provide additional knowledge to the model. this helps to rank the best answer at the top, with minimal increase in compute requirements and latency. previous research works @xcite have directly used existing pre-trained transformer encoders for contextual as2, by fine-tuning them on an input comprising of multiple sentences with different roles, i.e., the question, answer candidate, and context (previous and following sentences around the candidate). this structured input creates practical challenges during fine-tuning, as standard pre-training approaches do not align well with the downstream contextual as2 task, e.g., the language model does not know the role of each of these multiple sentences in the input. in other words, the extended sentence-level embeddings have to be learnt directly during fine-tuning, causing underperformance empirically. this effect is amplified when the downstream data for fine-tuning is small, indicating models struggling to exploit the context. in this paper, we tackle the aforementioned issues by designing three pre-training objectives that structurally align with the final contextual as2 task, and can help improve the performance of language models when fine-tuned for as2. our pre-training objectives exploit information in the structure of paragraphs and documents to pre-train the context slots in the transformer text input. we evaluate our strategies on two popular pre-trained transform-ers over five datasets. the results show that our approaches using structural pre-training can effectively adapt transformers to process contextualized input, improving accuracy by up to 8% when compared to the baselines on some datasets. ## related work answer sentence selection tanda @xcite established the sota for as2 using a large dataset (asnq) for transfer learning. other approaches for as2 include: separate encoders for question and answers @xcite , and compare-aggregate and clustering to improve answer relevance ranking @xcite . @xcite use lstms for answers and topics, improving accuracy for next sentence selection. @xcite use grus to model answers and local context, improving performance on two as2 datasets. @xcite propose a transformer encoder that uses context to better disambiguate between answer candidates. @xcite use unsupervised similarity matching techniques to extract relevant context for answer candidates from documents. ## contextual as2 as2 given a question q and a set of answer candidates @xmath0, . . . , s n }, the goal is to find the best s k that answers q. this is typically done by learning a binary classifier c of answer correctness by independently feeding the pairs (q, s i ), @xmath1, . . ., n} as input to c, and making c predict whether s i correctly answers q or not. at inference time, we find the best answer for q by selecting the answer candidate s k which scores the highest probability of correctness @xmath2, s i ). contextual as2 contextual models for as2 exploit additional context to improve the final accuracy. this has been shown to be effective (lauriola and moschitti, 2021) in terms of overcoming coreference disambiguation and lack of enough information to rank the best answer at the top. different from the above case, contextual as2 models receive as input a tuple (q, s i , c i ) where c i is the additional context. c i is usually the sentences immediately before and after the answer candidate. ## context-aware pre-training objectives we design a transformer pre-training task that aligns well with fine-tuning contextual as2 models, both structurally and semantically. we exploit the division of large corpora in documents and the subdivision of documents in paragraphs as a source of supervision. we provide triplets of text spans (a, b, c) as model inputs when pre-training, which emulates the structure of (q, s i , c i ) for contextual as2 models, where a, b and c play the analogous role of the question, the candidate sentence (that needs to be classified), and the context (which helps in predicting (a, b) correctness), respectively. formally, given a document d from the pre-training corpus, the task is to infer if a and b are two sentences extracted from the same paragraph @xmath3. given a question of the type "what are the acting roles of x", a standard lm can easily reason to select answers of the type "x acted/played in y", by matching the subject argument of the question with the object argument of the answer, for the same predicate acting/playing. however, the same lm would have a harder time selecting answers of the type "x appeared in y " because this requires learning the relation between the entire predicate argument structure of acting vs. the one of appearing. a lm pre-trained using the ssp task can learn these implications, as it reasons about concepts from s 3 , e.g., "appearing in prison break and just jordan" (which are tv series), being related to concepts from s 2 , e.g., "having an acting career" as the sentences belong to the same paragraph. the semantics learned by connecting sentences in the same paragraph transfer well downstream, as the model can re-use previously learned relations between entities and concepts, and apply them between question and answer candidates. relations in one sentence may be used to formulate questions that can be answered in the other sentence, which is most likely to happen for sentences in the same paragraph since every paragraph describes the same general topic from a different perspective. we design three ways of choosing the appropriate contextual information c for ssp. we present details on how we sample spans a, b and c from the pre-training documents below. ## datasets pre-training to perform a fair comparison and avoid any improvement stemming from additional pre-training data, we use the same corpora as roberta @xcite . this includes the english wikipedia, the bookcorpus @xcite , openwebtext @xcite and cc-news foot_0 . we pre-process each dataset by filtering away: (i) sentences shorter than 20 characters, (ii) paragraphs shorter than 60 characters and (iii) documents shorter than 200 characters. we split paragraphs into sequences of sentences using the nltk tokenizer @xcite and create the ssp pre-training datasets following section 4. refer appendix a.1 for more details. ## results public datasets on asnq, our pre-trained models get 3.8 -5.5% improvement in p@1 over the baseline using only the question and answer. our models also outperform the stronger contextual as2 baselines (1.6% with roberta and 2.4% with electra), indicating that our task-aware pre-training can help improve the downstream finetuning performance. on newsas2, we observe a similar trend, where all our models (except one) outperform both the standard and contextual baselines. on wikiqa, a smaller dataset, the contextual baselines under-performs the non-contextual baselines, highlighting that with few samples the model struggles to adapt and reason over three text spans. for this reason, our pre-training approaches provide the maximum accuracy improvement on wikiqa (up to 8 -9.1% over the non-contextual and contextual baselines). industrial datasets on iqad, we observe that the contextual baseline performs on par or lower than the non-contextual baseline, indicating that off-the-shelf transformers cannot effectively exploit the context available for this dataset. the answer candidates and context for iqad are extracted from millions of web documents. thus, learning from the context in iqad is a harder task than learning from it on asnq, where the context belongs to a single wikipedia document. our pre-trained models help to process the diverse and possibly noisy context of iqad, and produce a significant improvement in p@1 over the contextual baseline. combining the 3 ssp objectives we observe that combining all the objectives together does not always outperform the individual objectives, which is probably due to the misalignment between the different approaches for sampling context in our pre-training strategies. notice that we used a single classification head for all the three tasks, indirectly asking the model also to recognize the task to be solved among sdc, dpc or dslc. experiments with separate classification heads (one for each task) led to worse results in early experiments. choosing the optimal ssp objective our finetuning datasets have significantly different structures: asnq, newsas2 and wikiqa have answer candidates sourced from a single document (wikipedia for asnq and wikiqa and cnn daily mail articles for newsqa), while iqad has answer candidates extracted from multiple documents. this also results in the context for the former being more homogeneous (context for all candidates for a question is extracted from the same document), while for the latter the context is more heterogeneous (extracted from multiple documents for different answer candidates). our dpc and dslc pre-training approaches are well aligned in terms of the context that is used to help the ssp predictions. the former uses the remainder of the paragraph p as context (after removing a and b), while the latter uses the sentence previous and next to b in p . we observe empirically that the contexts for dpc and dslc often overlap partially, and are sometimes even identical (considering average length of paragraphs in the pre-training corpora is 4 sentences). this explains why models pre-trained using both these approaches perform comparably in on iqad, we observe that the sdc approach of providing context for ssp outperforms dpc and dslc. in sdc, the context c can potentially be very different from a and b (as it corresponds to the first paragraph of the document), and this can aid exploiting information and effectively ranking answer candidates from multiple documents (possibly from different domains) like for iqad. for these reasons, we recommend using dpc and dslc when answer candidates are extracted from the same document, and sdc when candidates are extracted from multiple sources. ## conclusion and future work in this paper, we have proposed three pre-training strategies for transformers, which (i) are aware of the downstream task of contextual as2, and (ii) use the document and paragraph structure information to define effective objectives. our experiments on three public and two industrial datasets using two transformer models show that our pre-training strategies can provide significant improvement over the contextual as2 models. in addition to local context around answer candidates (the previous and successive sentences), other contextual signals can also be incorporated to improve the relevance ranking of answer candidates. meta-information like document title, abstract/firstparagraph, domain name, etc. corresponding to the document containing the answer candidates can help answer ranking. these signals differ from the previously mentioned local answer context as they provide "global" contextual information pertaining to the documents for as2. our sdc objective, which uses the first paragraph of the document for the context input slot, captures global information pertaining to the document, and we hypothesise that this may improve downstream performance using other global contextual signals in addition to local answer context.