Dual Context-Guided Continuous Prompt Tuning for Few-Shot Learning (2022)

Abstract: Prompt-based paradigm has shown its competitive performance in many NLP tasks. However, its success heavily depends on prompt design, and the effectiveness varies upon the model and training data. In this paper, we propose a novel dual context-guided continuous prompt (DCCP) tuning method. To explore the rich contextual information in language structure and close the gap between discrete prompt tuning and continuous prompt tuning, DCCP introduces two auxiliary training objectives and constructs input in a pair-wise fashion. Experimental results demonstrate that our method is applicable to many NLP tasks, and can often outperform existing prompt tuning methods by a large margin in the few-shot setting.

## introduction with the rise of pretrained language models(plms), natural language processing(nlp) shifted from the fully-supervised paradigm to pretrain and fine-tune paradigms @xcite @xcite . to further utilize the large capacity of plms, a prompt-based paradigm is proposed to reformulate downstream tasks into an lm-like task upon the context and task-specific prompt. there are some issues with the prompt-based paradigm, especially prompt engineering. discrete prompts (a.k.a hard prompts) @xcite need expert-level experience to manually discover templates. to address this problem, automatic prompt design is conducted on gradient-based search @xcite , generation @xcite , ensembles @xcite and scoring @xcite . adapet @xcite provide a denser supervision during fine-tuning based on the label-conditioned language modeling task. however, these methods might get sub-optimal templates and require adequate validation data @xcite . what's more, it is unnecessary to limit prompts to hard-crafting text. continuous prompts (a.k.a soft prompts) @xcite take templates as additional trainable parameters. thus, prompt search can be simplified as optimizing parameters based on downstream task. recent works add layer-wise adaptive prompt parameters @xcite , data-dependent mixture @xcite and hard-soft hybrid prompt @xcite based on adequate training data. when it comes to the few-shot learning scenario, it remains unclear how to effectively learn continuous prompts. previous works mainly improve continuous prompts by additional prompt and target encoder @xcite @xcite . this paper presents a new model-agnostic perspective of further utilizing deep lm features. we propose a novel dual context-guided continuous prompt (dccp) tuning approach that makes plms better few-shot learners. our main concern is how to learn better continuous prompts with only a few samples, averting dependency on hand-craft engineering and large validation samples. considering that prompt-based models predict based on both prompt and context, the vanilla models learn about p (y |x context , h p rompt ). notably, additional prompt embeddings h prompt are optimized based on the given context x context with lm decoding task on the downstream target y in previous works. we give an insight into better continuous prompt tuning throughout the dual view of context-aware prompt and label-aware context representations. technically, we introduce a new label-aware context-masked input aligning with the vanilla context-aware prompt-masked input. we add two auxiliary training objectives for coupling layer-wise linguistic features. the dual view makes the model learn p (x context |y, h prompt ) and p (h context |y, h prompt ) throughout lm inner features, further optimizing the prompt embeddings. in a nutshell, dccp goes deep into plms representations for better continuous prompt tuning. we conduct experiments on 10 nlp datasets in the few-shot learning setting. dccp significantly outperforms conventional fine-tuning, discrete prompts, and previous works on continuous prompts. dccp achieves 89.6% (on average) of the full-supervised fine-tuning performance across all datasets with only 16 training samples. it obtains gain 11.8%, 2.5%, and 1.6% absolute improvement on average compared to conventional fine-tuning, vanilla continuous prompts @xcite , and state-of-the-art continuous prompts @xcite . we empirically demonstrate that dccp makes lm a better few-shot learner. ## methodology in this section, we first introduce the vanilla continuous prompt tuning model and then clarify our dual context-guided prompt tuning method. ## experiments we experiment our proposed architecture on 10 nlp tasks in the few-shot setting (@xmath0. the datasets involve sentiment analysis (sst-2, mr, cr), subjective analysis (subj), question type (trec), natural language inference (mnli, snli, qnli), paraphrase detection (mrpc, qqp). ## conclusion in this paper, we present a model-agnostic approach for advancing continuous prompt. specifically, we propose a novel dual context-guided continuous prompt tuning method for few-shot learning. our approach constructs a couple of dual inputs, including the origin context-aware prompt-masked input and label-aware context-masked one. then, we go deep into the language model to leverage linguistic features for two auxiliary constraints on the pairwise context inputs. the empirical results show that continuous prompts can be further revised during the procedure of reconstructing context.