Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2022)

Abstract: We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations, motivated by the intuition that the representation of a phrase in a passage should encode all questions that the phrase can answer in context. To this end, we train a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model on 80 million synthesized QA pairs. By encoding QA-relevant information, the bi-encoder’s token-level representations are useful for non-QA downstream tasks without extensive (or in some cases, any) fine-tuning. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection on four datasets, few-shot named entity recognition on two datasets, and zero-shot sentiment analysis on three datasets.

## introduction while masked language models build contextualized word representations, they are pre-trained with losses that minimize distance to uncontextualized word embeddings @xcite @xcite . this objective yields a good initialization for downstream fine-tuning, but the pre-trained representations themselves are not optimized for being immediately useful without fine-tuning. in this paper, we introduce question answering infused pre-training (quip), a new pretraining loss based on question answering (qa) that depends much more directly on context. quip learns improved token-level representations that are useful in zero-shot and few-shot settings, where extensive fine-tuning is not possible. our intuition for quip is that the contextualized representation for a phrase in a passage should contain enough information to identify all the questions the violin concerto in d major, op. 77, was composed by johannes brahms in 1878 and dedicated to his friend, the violinist joseph joachim. what did brahms write in 1878? what was dedicated to joachim? who was friends with joachim? who played the violin? who wrote the violin concerto? figure 1 : an overview of question answering infused pre-training. our model independently creates vector representations (middle) for phrases in a passage (top) and for synthesized questions (bottom). our objective encourages the vector for each phrase to have high similarity with the vectors for all questions it answers. bi-encoder qa has been used before for efficient open-domain qa via phrase retrieval @xcite @xcite , but its lower accuracy compared to cross-encoder qa has previously been viewed as a drawback. we instead view the relative weakness of bi-encoder qa as an opportunity to improve contextual representations via knowledge distillation, as self-training can be effective when the student model must solve a harder problem than the teacher @xcite . in particular, since the bi-encoder does not know the question when encoding the passage, it must produce a single passage representation that simultaneously encodes the answers to all possible questions. in contrast, while cross-encoder qa models are more accurate, they depend on a specific question when encoding a passage; thus, they are less suited to downstream use cases that require contextualized representations of passages in isolation. we show that quip token-level representations are useful in a variety of zero-shot and few-shot learning settings, both because the representations directly encode useful contextual information, and because we can often reduce downstream tasks to qa. for few-shot paraphrase detection, quip with bertscore-based features @xcite outperforms prior work by 9 f1 points across four datasets. for few-shot named entity recognition (ner), quip combined with an initialization scheme that uses question embeddings improves over roberta-large by 14 f1 across two datasets. finally, for zero-shot sentiment analysis, quip with question prompts improves over roberta-large with mlm-style prompts by 5 accuracy points across three datasets, and extracts interpretable rationales as a side effect. through ablations, we show that using real questions, a strong teacher model, and the bi-encoder architecture are all crucial to the success of quip. other design decisions (e.g., question generation decoding strategies) do not qualitatively affect our main findings, pointing to the stability of the quip approach. 1 1 code to reproduce all results can be found at https: //github.com/facebookresearch/quip . ## qa infused pre-training qa infused pre-training (quip) involves pretraining contextual representations with a biencoder extractive qa objective. in contrast with masked language modeling, quip's training objective directly encourages contextual representations to encode useful semantic information, namely information about what questions can be answered by each span. in contrast with a cross-encoder qa model, quip's bi-encoder is trained to encode single passages rather than passage-question pairs, making it more transferable to tasks involving single passages. moreover, quip learns to push each phrase's representation far away from those of questions the phrase does not answer; this ability to represent unanswerability is crucial for correctly handling some question-based prompts. we now introduce some basic notation ( §2.1), then describe the quip pipeline, which consists of three steps: question generation ( §2.2), crossencoder teacher re-labeling ( §2.3), and bi-encoder training ( §2.4). ## downstream tasks we evaluate quip on zero-shot paraphrase ranking, few-shot paraphrase classification, few-shot ner, and zero-shot sentiment analysis. different tasks showcase different advantages of quip. for paraphrase detection and ner, quip succeeds by learning meaningful token-level contextualized representations for single passages, whereas mlm representations are trained to reconstruct uncontextualized word embeddings, and the cross-encoder qa model is trained to represent passage-question pairs. for ner and sentiment analysis, we prompt quip with questions, leveraging its question-answering abilities. compared with a cross-encoder, quip's bi-encoder architecture enables a more efficient way to use question prompts in ner, and yields more reliable scores when questions are unanswerable in sentiment analysis. we focus on zero-shot and few-shot settings, as these require pre-trained models that are useful without fine-tuning on a large task-specific training dataset. quip addresses this need by anticipating what information might be useful for downstream tasks-namely, information found in question-answer pairs. ## discussion and related work we build on work in question generation and answering, pre-training, and few-shot learning. ## conclusion in this work, we pre-trained token-level contextual representations that are useful for downstream fewshot learning. our key idea was to use questionanswer pairs to define what information should be encoded in passage representations. we showed that these representations are useful for a variety of standard nlp tasks in zero-and few-shot settings, including paraphrase detection, named entity recognition, and sentiment analysis, across nine total datasets. looking forward, we hope to see more work on designing pre-training objectives that align with downstream needs for few-shot learning. we only applied calibration for the main quip model, as we did not find calibration to improve results for either lm-bff or the cross-encoder qa student model.