KECP : Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering (2022)

Abstract: Extractive Question Answering (EQA) is one of the most essential tasks in Machine Reading Comprehension (MRC), which can be solved by fine-tuning the span selecting heads of Pre-trained Language Models (PLMs). However, most existing approaches for MRC may perform poorly in the few-shot learning scenario. To solve this issue, we propose a novel framework named Knowledge Enhanced Contrastive Prompt-tuning (KECP). Instead of adding pointer heads to PLMs, we introduce a seminal paradigm for EQA that transforms the task into a non-autoregressive Masked Language Modeling (MLM) generation problem. Simultaneously, rich semantics from the external knowledge base (KB) and the passage context support enhancing the query’s representations. In addition, to boost the performance of PLMs, we jointly train the model by the MLM and contrastive learning objectives. Experiments on multiple benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in few-shot settings by a large margin.

## introduction span-based extractive question answering (eqa) is one of the most challenging tasks of machine reading comprehension (mrc). a majority of recent approaches @xcite @xcite add pointer heads @xcite to pre-trained language models (plms) to predict the start and the end positions of the answer span (shown in figure 1(a) ). yet, these conventional fine-tuning frameworks heavily recently, a branch of prompt-based fine-tuning paradigm (i.e. prompt-tuning) arises to transform the downstream tasks into the cloze-style problem @xcite @xcite @xcite @xcite . to specify, task-specific prompt templates with @xcite tokens are added to input texts @xcite denotes the masked language token in plms). the results of the masked positions generated by the mlm head are used for the prediction foot_0 . by prompt-tuning, we can use few training samples to fast adapt the prior knowledge in plms to downstream tasks. a natural idea is that we can transform eqa into the mlm task by adding a series of masked language tokens. as shown in figure 1 (b), the query is transformed into a prompt template containing multiple [mask] tokens, which can be directly used for the answer tokens prediction. however, we observe that two new issues for vanilla plms: 1) the mlm head, which is based on single-token non-autoregressive prediction, has a poor inference ability to understand the task paradigm of eqa ; 2) there are many confusing span texts in the passage have similar semantics to the correct answer, which can unavoidably make the model produce negative answers. therefore, a natural question arises: how to employ prompttuning over plms for eqa to achieve high performance in the few-shot learning setting? in this work, we introduce kecp, a novel knowledge enhanced contrastive prompting framework for the eqa task. we view eqa as an mlm generation task that transform the query to a prompt with multiple masked language tokens. in order to improve the inference ability, for each given example, we inject related knowledge base (kb) embeddings into context embeddings of the plm, and enrich the representations of selected tokens in the query prompt. to make plms better understand the span prediction task, we further propose a novel span-level contrastive learning objective to boost the plm to distinguish the correct answer with the negatives with similar semantics. during the inference time, we implement a highlyefficient model-free prefix-tree decoder and generate answers by beam search. in the experiments, we evaluate our proposed framework over seven eqa benchmarks in the few-shot scenario. the results show that our method consistently outperforms state-of-the-art approaches by a large margin. specifically, we achieve a 75.45% f1 value on squad2.0 with only 16 training examples. to sum up, we make the following contributions: ## related work in this section, we summarize the related work on eqa and prompt-tuning for plms. ## the kecp framework in this section, we formally present our task and the techniques of the kecp framework in detail. the overview of kecp is shown in figure 2 . ## experiments in this section, we conduct extensive experiments to evaluate the performance of our framework.