Paragraph-based Transformer Pre-training for Multi-Sentence Inference (2022)

Abstract: Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. Recent studies show that these tasks benefit from modeling dependencies across multiple candidate sentences jointly. In this paper, we first show that popular pre-trained transformers perform poorly when used for fine-tuning on multi-candidate inference tasks. We then propose a new pre-training objective that models the paragraph-level semantics across multiple input sentences. Our evaluation on three AS2 and one fact verification datasets demonstrates the superiority of our pre-training technique over the traditional ones for transformers used as joint models for multi-candidate inference tasks, as well as when used as cross-encoders for sentence-pair formulations of these tasks.

## introduction pre-trained transformers @xcite @xcite have become the de facto standard for several nlp applications, by means of fine-tuning on downstream data. the most popular architecture uses self-attention mechanisms for modeling long range dependencies between compounds in the text, to produce deep contextualized representations of the input. there are several downstream nlp applications that require reasoning across multiple inputs candidates jointly towards prediction. some popular examples include (i) answer sentence selection (as2) @xcite , which is a question answering (qa) task that requires selecting the best answer from a set of candidates for a question; and (ii) fact verification @xcite , which reasons whether a claim is supported/refuted by multiple evidences. inherently, these tasks can utilize information from multiple candidates (answers/evidences) to support the prediction of a particular candidate. pre-trained transformers such as bert are used for these tasks as cross-encoders by setting them as sentence-pair classification problems, i.e, aggregating inferences independently over each candidate. recent studies @xcite have shown that these tasks benefit from encoding multiple candidates together, e.g., encoding five answer candidates per question in the transformer, so that the cross-attention can model dependencies between them. however, zhang et al. only improved over the pairwise cross-encoder by aggregating multiple pairwise cross-encoders together (one for each candidate), and not by jointly encoding all candidates together in a single model. in this paper, we first show that popular pretrained transformers such as roberta perform poorly when used for jointly modeling inference tasks (e.g., as2) using multi-candidates. we show that this is due to a shortcoming of their pre-training objectives, being unable to capture meaningful dependencies among multiple candidates for the finetuning task. to improve this aspect, we propose a new pre-training objective for 'joint' transformer models, which captures paragraph-level semantics across multiple input sentences. specifically, given a target sentence s and multiple sentences (from the same/different paragraph/document), the model needs to recognize which sentences belong to the same paragraph as s in the document used. joint inference over multiple-candidates entails modeling interrelated information between multiple short sentences, possibly from different paragraphs or documents. this differs from related works @xcite @xcite that reduce the asymptotic complexity of transformer attention to model long contiguous inputs (documents) to get longer context for tasks such as machine reading and summarization. we evaluate our pre-trained multiple-candidate based joint models by (i) performing as2 on asnq @xcite , wikiqa @xcite , trec-qa @xcite datasets; and (ii) fact verification on the fever @xcite dataset. we show that our pre-trained joint models substantially improve over the performance of transformers such as roberta being used as joint models for multi-candidate inference tasks, as well as when being used as cross-encoders for sentence-pair formulations of these tasks. ## experiments we evaluate our joint transformers on three as2 and one fact verification datasets foot_0 . common lm benchmarks, such as glue @xcite , are not suitable for our study as they only involve sentence pair classification. ## conclusions in this paper we have presented a multi-sentence cross-encoder for performing inference jointly on multiple sentences for tasks like answer sentence selection and fact verification. we have proposed a novel pre-training task to capture paragraph-level semantics. our experiments on three answer selection and one fact verification datasets show that our pre-trained joint models can outperform pairwise cross-encoders and pre-trained lms when directly used as joint models.