Hard Sample Aware Prompt-Tuning (2023)

Abstract: Prompt-tuning based few-shot learning has garnered increasing attention in recent years due to its efficiency and promising capability. To achieve the best performance for NLP tasks with just a few samples, it is vital to include as many informative samples as possible and to avoid misleading ones. However, there is no work in prompt-tuning literature addressing the problem of differentiating informative hard samples from misleading ones in model training, which is challenging due to the lack of supervision signals about the quality of the samples to train a well-performed model. We propose a Hard Sample Aware Prompt-Tuning framework (i.e. HardPT) to solve the non-differentiable problem in hard sample identification with reinforcement learning, and to strengthen the discrimination of the feature space without changing the original data distribution via an adaptive contrastive learning method. An extensive empirical study on a series of NLP tasks demonstrates the capability of HardPT in few-shot scenarios. HardPT obtains new SOTA results on all evaluated NLP tasks, including pushing the SST-5 accuracy to 49.5% (1.1% point absolute improvement), QNLI accuracy to 74.6% (1.9% absolute improvement), NMLI accuracy to 71.5 (0.7% absolute improvement), TACREV F_1 -score to 28.2 (1.0 absolute improvement), and i2b2/VA F_1 -score to 41.2 (1.3 absolute improvement).

## introduction in recent years, self-supervised pre-trained language models (plms) like gpt @xcite , and bert @xcite have gained significant popularity in various natural language * equal contribution. † work done during yuanjian xu and qi an's internship at air, tsinghua university. to achieve the best performance for nlp tasks with just a few samples, it is vital to include as many informative samples as possible and to avoid misleading ones. however, there is no work in the literature addressing the problem of differentiating informative hard samples from misleading ones in prompt-tuning. the concept of hard samples has been discussed in the field of computer vision for object detection and image classification tasks @xcite . during the training process, high-loss samples are typically selected as hard samples, and their weights are adjusted through resampling or modifying the loss. however, relying solely on the loss function results in poor performance in identifying misleading samples. additionally, this approach can result in a drift of the original data distribution and overlook the optimization of the sample feature space. most recently, @xcite proposes an approach named ehn which is specifically designed for distinguishing hard samples from noisy samples in the context of histopathology image classification tasks. however, this method relies on additional prior knowledge as supervision signals, making it impractical for real-world applications. automatically differentiating misleading and hard samples in nlp tasks is challenging due to the lack of supervision signals about the quality of the samples to train a quality classification model. another work, dataset cartography (dc) @xcite aims to mine the statistical metrics, confidence, and variability, to evaluate the quality of samples in nlp datasets. however, dc fails to differentiate between misleading and hard samples, categorizing them both as "hard-to-learn". accurately estimating the quality of nlp samples, particularly non-committal language descriptions, remains challenging. to address the problem of differentiating informative hard samples from misleading ones in prompt-tuning, we propose a framework named hard sample aware prompt-tuning (hardpt). we classify the samples into three categories: easy, hard, and misleading samples. easy samples mean they are easily classified correctly by the model, while the hard samples are meant to be difficult for the model to learn correctly. misleading samples refer to harmful samples during data annotation. due to the lack of supervised signals for sample quality, we design a reinforcement learning network to solve the non-differentiable problem in hard sample identification. to better leverage the identified hard samples, we innovatively propose an adaptive contrastive learning method to strengthen the discrimination of the feature space without changing the original data distribution. we conduct an extensive empirical study on various nlp tasks to demonstrate the capability of hardpt in few-shot scenarios. remarkably, hardpt achieved state-of-the-art (sota) results across all evaluated nlp tasks, including pushing the sst-5 accuracy to 49.5% (1.1% absolute improvement), improving the qnli accuracy by 1.9 percentage points, resulting in a noteworthy 74.6%. moreover, hardpt improves the nmli accuracy to reach 71.5 (0.7% absolute improvement), tacrev f 1 -score to 28.2 (1.0 absolute improvement), and i2b2/va f 1 -score to 41.2 (1.3 absolute improvement). these results highlight the exceptional performance of hardpt in the few-shot scenarios. our key contributions can be summarized as follows: prompt-tuning: prompt-tuning @xcite @xcite @xcite @xcite is an efficient way to adapt pretrained language models (plms) to downstream tasks without tuning the parameters of plms. depending on the type of prompt, prompt-tuning is divided into two categories: soft prompt and hard prompt. soft prompt leverage trainable parameters as prompts @xcite @xcite , while hard prompt employs natural language strings as prompts. with the emergence of gpt-3 (brown et al., 2020), the hard prompt has gained significant attention in recent years, particularly in the context of few-shot learning. and intensive efforts have been devoted to improving the prompts. zong et al. (2021) utilize demonstrations to enhance prompt-tuning in few-shot scenarios and achieve improvement on various nlp datasets. liang et al. (2022) introduce more demonstrations and utilize contrastive learning to compare different demonstrations on the same datasets. han et al. (2021) propose to construct prompts automatically by combining sub-prompts with logic rules. however, the impact of the samples remains underexplored in prompt-tuning. contrastive learning: contrastive learning (cl) @xcite @xcite is an effective method for representation learning that brings samples of the same class closer together while pushing those of different classes apart in the representation space. cl can be divided into unsupervised cl and supervised cl according to whether the pretext task requires labeled data. unsupervised cl has gained widespread popularity because it reduces the need for labeled data for the model. @xcite utilize cl by generating two distinct augmented versions of the same sentence. they employ four methods as a data augmentation module at the embedding layer. this approach effectively leverages ## method previous research has demonstrated the effectiveness of prompt-tuning as a practical approach to address few-shot learning in nlp. however, many existing methods only focus on categorizing data into easy and hard samples, overlooking the crucial distinction between hard samples and mislabeled misleading samples. in the context of few-shot learning, failure to distinguish between these two types of samples can lead to underutilization of the value inherent in hard samples and result in a decline in model performance. therefore, it is imperative to address this issue to fully leverage the potential of hard samples in the few-shot scenario. to illustrate and address these problems more clearly, this section begins by introducing the fundamental paradigm of prompt-tuning in section 3.1. section 3.2 describes the hardpt algorithm, which mainly includes the detection and utilization methods of hard samples. towards the end of this section, we outline the training process of hardpt. ## experiment in this section, we present the experimental setup and results, which are divided into three parts. in section 4.1, we provide a brief description of the public datasets utilized in this paper. the baseline models employed in the experiments are outlined in section 4.2. in section 4.3, we present the results and provide a detailed analysis. for additional information regarding the dataset, experimental conditions, and hyperparameter settings, please refer to appendices a to c. ## analysis to validate the effectiveness of each component in hardpt and its robustness against noise, we conducted several controlled experiments. firstly, we conducted ablation experiments to verify the impact of hard sample identification and the sample augmentation method in contrastive learning. secondly, we performed experiments to assess the algorithm's robustness in the presence of noise. ## conclusions this paper proposes hardpt, the first prompttuning framework for hard sample identification and utilization. hardpt focuses on the influence of sample quality on the model in the few-shot scenario based on prompt-tuning. our method can distinguish hard samples from misleading samples without data quality labels, and mine the information contained in hard samples using contrastive learning based on the features of hard samples. an extensive empirical study on a series of nlp tasks demonstrates the capability of hardpt in few-shot scenarios. hardpt obtains new sota results on all evaluated nlp tasks, including pushing the sst-5 accuracy to 49.5% (1.1% point absolute improvement), qnli accuracy to 74.6% (1.9% absolute improvement), nmli accuracy to 71.5 (0.7% absolute improvement), tacrev f 1 -score to 28.2 (1.0 absolute improvement), and i2b2/va f 1 -score to 41.2 (1.3 absolute improvement).