Knowledge Stimulated Contrastive Prompting for Low-Resource Stance Detection (2022)

Abstract: Stance Detection Task (SDT) aims at identifying the stance of the sentence towards a specific target and is usually modeled as a classification problem. Backgound knowledge is often necessary for stance detection with respect to a specific target, especially when there is no target explicitly mentioned in text. This paper focuses on the knowledge stimulation for low-resource stance detection tasks. We firstly explore to formalize stance detection as a prompt based contrastive learning task. At the same time, to make prompt learning suit to stance detection, we design a template mechanism to incorporate corresponding target into instance representation. Furthermore, we propose a masked language prompt joint contrastive learning approach to stimulate the knowledge inherit from the pre-trained model. The experimental results on three benchmarks show that knowledge stimulation is effective in stance detection accompanied with our proposed mechanism.

## introduction stance detection is one of the most challenging nlp tasks, which aims to identify the stance of a piece of text towards a given target. recent years have witnessed rapid progress on learning an intent classification model for target stance @xcite @xcite .though such models in advanced neural architectures @xcite are capable of replying with responses regarding to specific target in a piece of text, to exactly detect the stance of target, background knowledge is often necessary, especially when there is no target explicitly mentioned in text. take the tweet "i'm confused to hear that trump win the poll today." as example, the stance detection system should detected the attitude "against" when given target "trump". however, when target is "biden", the stance detected should became "favour" correspondingly. furthermore, deep neural networks often require large-scale highquality labeled training data to achieve state-of-theart performance @xcite . however, the cost of labeled data collection is not trival. in this paper, we study the low-resource stance detection tasks, including target-specific stance detection and cross-target stance detection tasks, where only small labeled data is available. previous works often introduce external data for the models to learn. @xcite propose to utilize external training knowledge data which could be expensive to obtain. to solve the above dilemma, many existing stance detection works @xcite resort to fine tuning pre-trained language models (plms). however, we argue that, in the low-resource stance detection tasks, directly finetuning all parameters of plms with small training data (especially when there are less than 100 samples) could result in over-fitting and plms simply memorizes the training instances. recently, several works propose prompt tuning @xcite , which only backpropagates the error to soft prompts (i.e., a sequence of continuous vectors prepended to the input of plms) instead of the entire model. they show that prompt tuning is sufficient to be competitive with full model tuning while significantly reducing the amount of parameters to be tuned. thus, the prompt tuning is quite suitable to tackle the above over-fitting issue in low-resource finetuning. unfortunately, we find a major limitation of taskbased prompts that the prompts are too coarsegrained and ignore fine-grained information in input data. in state-of-the-art methods, all input data within a tuning task shares an identical prompt implying the task domain. however, besides the task, the specific content in the input data also contains context that can help the plm retrieve more relevant knowledge, for example, the specific object being talked about. such knowledge behinds the input data should be fully exploited to unleash the potential of prompts. the key is that there is a gap between prompt and pre training, because the template in prompt is absent during pre training. in order to make up for this defect, we propose to use cloze task that more familiar with plm, that is, only do the mask method in the original input to stimulate the richer knowledge learned by the original plms. to make plms better understand the stance detection task, we further propose a novel contrastive learning objective to boost the plm to distinguish the different stance. we conduct experiments with three benchmarks of knowledge-grounded stance detection that are constructed by crowd-sourcing. evaluation results in terms of both automatic metrics and human judgment indicate that our model not only achieves comparable performance with the state-of-the-art model that is learned from crowd-sourced training sets, even with a small amount of data, our model surpasses the performance of the state-of-the-art model that using whole data and external knowledge, but also exhibits a good generalization ability over different targets and different datasets. our contributions are four-fold: (1) exploration of knowledge-grounded stance detection under a low-resource setting; (2) proposal of a knowledge stimulated method that bridge the gap between plm and prompt tuning; (3) proposal of a unified cloze contrastive prompting learning approach; and (4) empirical verification of the effectiveness of the proposed approach on three benchmarks of knowledge-grounded stance detection. prompt-tuning for plms there is an emerging interest in using prompts to extract knowledge from large language models @xcite @xcite @xcite . prompt-tuning is a signicant research in the past years. the gpt-3 @xcite fueled the development of prompt-based learning, which relies on handcraft prompts and achieves impressive per-formance. to further construct automatic prompt, autoprompt @xcite and lm-bff @xcite propose automatic prompt construction through generate discrete tokens. recently, prefix-tuning @xcite , p-tuningv2 @xcite , ptr @xcite , and soft-prompts @xcite propose continuous prompt construction, which introduce differential parameters to solve the issue. unlike previous work @xcite , our prompt based framework mainly focuses on sdt, which is the novel exploration of this challenging task in low resource setting. contrastive learning contrastive learning aims to learn representations with self-supervision, so that similar samples are embedded close to each other (positive pair) while pushing away samples that are dissimilar (negative pairs). such representations have been shown to capture the semantic information of the samples by maximizing the lower bound of the mutual information between two augmented views @xcite . several methods for contrastive learning have been developed so far @xcite @xcite . supcon @xcite ) is a spe- ## method heading for learning an effective knowledge stimulation model for the task of knowledge-grounded stance detection, we need to deal with several challenges: (1) how to stimulate knowledge without inject external data in stance detection; (2) how to stimulate knowledge with low-resource data. this section first formulates our proposed knowledge stimulated contrastive prompting method (kscp) for low-resource stance detection task. we then introduce the three important components in kscp, including i) prompt-based learning in pre-trained language models; ii) word cloze task and iii) contrastive learning. figure 1 shows the overall of kscp. ## conclusion in this paper, we present the first prompt-based model kscp for low-resource stance detection. experiments on three benchmarks show the effectiveness of our proposed kscp method. in the future, we plan to expand kscp to other nlp tasks and other settings, including question answering, machine reading comprehension and text generation tasks.