P ro QA : Structural Prompt-based Pre-training for Unified Question Answering (2022)

Abstract: Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.

## introduction question answering has long been an inspirational challenge in nlp research, and is viewed as the next-generation search engine and an essential tool for human beings to obtain knowledge @xcite . many distinct datasets @xcite @xcite have been proposed along with the research trend on qa, involving very diverse question types (e.g., extractive qa, abstractive qa, multiple-choice qa), domains (e.g., finance, daily events), and answer types (e.g., free-formed text, selected option). the majority of previous works focus on tasks with specific question types @xcite @xcite or specific domains @xcite . recent research on large pre-trained language models @xcite indicates that there may be tight connections among various tasks, which sheds light on a unified paradigm that can be potentially applied to solve various qa tasks to model their commonality. this observation motivates us to develop a unified qa model, which can model both the commonly-required qa ability and the difference between various qa tasks within a same paradigm. to achieve this goal, there are several key challenges needed to be addressed: (1) how to model commonalities and enhance transferability among different qa tasks in various domains/formats while reducing the conflict between them? (2) how to construct large-scale qa corpus as the highquality qa-centric data is scarce for pre-training? in light of this, we conceive proqa, a unified qa paradigm, which builds up a general model to solve different qa tasks utilizing a structural prompt and improves commonly-required qa ability via structural prompt-based pre-training. firstly, to model the commonalities and distinguish task differences, we adopt a structural prompt to organize the inputs with a unified structurally designed input schema. as illustrated in fig. 1 , given the complex components (e.g., "domain", "format", "task", "question", "passage") as inputs, proqa divides components into multiple key-value pairs, in which a specific component like "question" denotes a key, and the specific in- secondly, to alleviate data sparsity problem and empower the model with transferability to the adaptation of new tasks, we conduct structural promptbased pre-training. we first build a large-scale synthetic qa corpus automatically from wikipedia, utilizing only a few seed datasets as the prior supervisions for pre-training corpus construction and finally covering primary qa formats. then we format the pre-training data with the structural prompt, and teach the model to learn the general purpose qa-centric ability and the functionality of each component in the structural prompt via pre-training. we evaluate the effectiveness of proqa on 11 downstream qa benchmarks, and the results show that our system achieves consistent performance boost in full data fine-tuning, few-shot learning, and zero-shot learning settings. experiments demonstrate that proqa can better mitigate the catastrophic forgetting issue during continual learning by restoring the task-specific soft prompts residing in the structural prompt. further analyses illustrate that our model has better transferability as it can be more quickly adapted to a newly involved task. ablation studies verify the effectiveness of both the soft prompt and prompt-based pre-training. the contributions are summarized as follows: • we propose proqa, a unified qa framework for solving various tasks within a single paradigm, taking an extensible and learnable structural prompt as the bridge. to this end, inspired by the success of casting multiple tasks into the same text-to-text format @xcite , researchers propose to learn a single model to unify various qa formats, alleviating the labor of taskspecific designs @xcite . however, these models (1) do not explicitly model the task or component characteristics, thus failing to properly disentangle the difference among qa tasks; and (2) overly rely on supervised data from specific tasks, which may not be available under data-scarce scenarios. qa-centric pre-training. numerous efforts have been spent on improving plms' reasoning abilities with an intermediate pre-training stage before fine-tuning on target qa tasks, including (1) language modeling adaptation with salient span masking, which trains plms to recover randomly chosen @xcite or machine-generated @xcite masked named entities in the raw corpus; (2) training data augmentation @xcite with synthetic question-answer-context triples, such as generating (a) pseudo questions through adversarial training @xcite , knowledge bases @xcite or machine translation @xcite , (b) pseudo answers exploiting recurring spans @xcite or rules based on heuristics @xcite and (c) pseudo contexts via information retrieval @xcite . nevertheless, these works largely target at improving a certain reasoning ability for plms, and thus cannot be easily generalized to other qa tasks. prompts for plms. to effectively stimulate the knowledge acquired through pre-training, prompt-oriented fine-tuning is receiving increasing attention @xcite , which re-formulates the objective of downstream tasks similar to that of pre-training by inserting manually designed @xcite or automatically searched @xcite hard prompt tokens into the input text. considering that discrete prompts may not be an optimal solution in the continuous embedding space, recent works @xcite proposed tunable soft prompts. it achieves satisfying performance especially when the model size grows extremely large @xcite . compared with the cumbersome parameters in plms, soft prompts are lightweight and pluggable, which paves the way for our goal of flexible adaptation to a new qa task. ## proqa in this section, we detailedly describe the whole framework of proqa for general purpose qa, which solves various qa tasks within the same paradigm. ## discussion in this section, we discuss on how to extend the proqa to a new task even with a new schema, and sheds light on potential future directions. 1) task adaptation with structural prompt: the design of structural prompt empowers proqa with better expandability. in our main experiments, we adopt 3 format types and 11 qa tasks. in the future, we can adapt proqa to more tasks, formats, domains, and new input schema. intuitively, when being adapted to a new task with unseen format/domain, proqa can initialize the specific soft prompts and learn the characteristic of the new domain/task through model training. moreover, if we encounter a new input schema that involves new keys (e.g., "extracted entities or commonsense knowledge"), we can add a new key-value pair in the input schema and learns the functionality of the new key indicator through training. ## conclusion we introduce proqa, a unified qa paradigm that adopts a single model for solving various qa tasks with the bridge of a structural prompt. structural prompt simultaneously models the common ability required for various tasks and keeps the speciality of each task, through a structurally designed learnable input schema. we further conduct structural prompt-based pre-training, seeking to empower the model with general qa-centric ability and injects the semantic knowledge of the structural prompt into the pre-training model. experimental results on 11 qa benchmarks demonstrate that proqa can significantly boost performance on all settings. further analyses show that our method can better mitigate the catastrophic forgetting issue during continual learning, and our method can be adapted to a newly involved task more quickly, by taking the advantages of the structural prompt. in the future, we hope our analysis could inspire more explorations on the unified qa methods, or the unification of distinct tasks with complex inputs modeling by the structural prompt. we also hope structural prompt can be further utilized into the unification of more tasks with complex inputs.