EICO : Improving Few-Shot Text Classification via Explicit and Implicit Consistency Regularization (2022)

Abstract: While the prompt-based fine-tuning methods had advanced few-shot natural language understanding tasks, self-training methods are also being explored. This work revisits the consistency regularization in self-training and presents explicit and implicit consistency regularization enhanced language model (EICO). By employing both explicit and implicit consistency regularization, EICO advances the performance of prompt-based few-shot text classification. For implicit consistency regularization, we generate pseudo-label from the weakly-augmented view and predict pseudo-label from the strongly-augmented view. For explicit consistency regularization, we minimize the difference between the prediction of the augmentation view and the prediction of the original view. We conducted extensive experiments on six text classification datasets and found that with sixteen labeled examples, EICO achieves competitive performance compared to existing self-training few-shot learning methods.

## introduction recently, @xcite proposed a cloze-style few-shot learning method, pet. by filling the gap between pre-training and fine-tuning with prompt and verbalizer, pet achieved competitive performance with @xcite with smaller language models like bert @xcite , t5 @xcite and gpt-2 @xcite . however, the lack of labeled data still limits the performance of few-shot learning. although acquiring labeled data is costly, unlabeled data is relatively easy to obtain. leveraging unlabeled data to improve the performance of the few-shot language model through semi-supervised learning is a promising way. in this paper, we focus on advancing the performance of the few-shot language model via semi-supervised learning. semi-supervised learning is a method that can leverage both labeled and unlabeled data. a common yet effective semi-supervised method is selflearning, which uses a trained teacher model to generate pseudo-labels for unlabeled examples, and then trains a student model on both labeled and pseudo-labeled examples to utilize the domainrelevant information contained in the unlabeled data. fixmatch @xcite and its natural language processing adaption sflm @xcite are robust self-training implementations, which generate pseudo-label on the weaklyaugmented view of an example and predict pseudolabel by the strongly-augmented view. however, the implicit consistency regularization introduced by the above training scheme may be suboptimal since class distribution information is lost in pseudo-label generation. to address the above problem, we propose explicit and implicit consistency regularization enhanced language model (eico). beside implicit consistency regularization, eico utilizes explicit consistency regularization by minimizing the difference between the prediction of the augmented view and the prediction of the original view. to validate the effectiveness and robustness of eico, we conduct extensive experiments on six natural language understanding tasks. the result of our experiments confirms that eico can be leveraged to improve the performance of the few-shot text classification. moreover, we find that eico presents robustness among different low resources situations and different model size by ablation study. ## conclusion in this work, we propose eico, a simple yet effective self-training prompt-based few-shot text classification method, where explicit consistency regularization is provided by the agreement of the augmented views of example, and implicit consistency regularization by the pseudo-label technique are utilized. we conducted comprehensive experiments over six text classification datasets and found that eico outperformed existing methods. moreover, in the ablation study, we explore the impact of the number of labeled examples k and two different model sizes and found that eico can consistently achieve competitive performance.