CCP refix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification (2024)

Abstract: Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, inspired by the human-decision process that the most ambiguous classes would be mulled over for an instance, we propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification. We conduct experiments on many-class benchmark datasets in both the fully supervised setting and the few-shot setting, which indicates that our model outperforms former baselines.

## introduction while the fine-tuning approach has been highly successful in the field of natural language processing, enabling the effective application of knowledge to specific tasks, a significant disparity still exists between the pre-training and fine-tuning stages. this figure 1: an illustrative example of entity typing task from fewnerd @xcite dataset. option a is its ground-truth label, and option b is the counterfactual. red words are the related attributes for the question. to effectively utilize masked language models (mlms) in prompt tuning, it's essential to create a task-specific template and verbalizers, forming a cloze-style task. typically, the template might be a natural language prompt or a sequence of continuous tokens to engage the language model. the verbalizers, on the other hand, are often phrases in natural language that correspond to specific task labels. for example, in natural language inference (nli), a training example could be structured with a prompt like " @xcite [mask] [hypothesis]". in this setup, a group of label words is crafted as potential options to fill the placeholder (e.g., @xcite ) in the designed template. again, in nli, the verbalizers are defined as {then, maybe and but}, corresponding the three-class categories {entailment, neural and contradiction}. clearly, it is quite feasible for experts to choose appropriate label words, given that there are distinct semantic boundaries between these mutually exclusive labels. as the number of labels increases, the semantic distinctions among many categories can become blurred, potentially leading to overlaps and resulting in the issue of verbalizer ambiguity. this phenomenon is highlighted in studies such as @xcite ; @xcite , which note the high sensitivity of performance to the selection of label words. for example, consider the entity typing task, where categories like "person-actor" and "person-employee" both fall under the same broader category of "person", as illustrated in fig 1 . to address this issue of verbalizer ambiguity, @xcite proposed manually creating logic rules to combine multiple sub-prompts into a final prompt for each class. however, this approach is limited due to the need for time-consuming and expert-devised logic rules. taking inspiration from the social science research @xcite , we adopt the contrastive procedure of human explanation to generate diverse information prefixes for training instances. concretely, rather than explaining "why a", it is more effective to explain "why a not b", where b serves as an implicit counterfactual of a within the current context. in figure 1 , we present an instance from the fewnerd @xcite dataset, where the task is to classify the type associated with greg. from a machine learning perspective, a welltrained model will recognize that greg is associated with multiple attributes, including "houston", "company" and "actor", all of which are deemed valuable for prediction. as illustrated in figure 1 , these contributed attributes can be redundant for prediction as highlighting. hence, the contrastive explanation approach tends to overlook most similarity attributes between "employee" and "actor", focusing instead on the more salient semantics that are critical for the model's differentiation task. in this paper, we propose counter-factual contrastive prefix-tuning, or ccprefix foot_0 , designed to reduce semantic vagueness among verbalizers and address the issue of verbalizer ambiguity. our process begins by constructing all possible factcounterfactual label pairs, with each class alternately assumed as the fact while the other classes are treated as counterfactuals. each instance is then projected onto the subspaces spanned by these factcounterfactual pairs, generating a range of potential contrastive attributes. these potential attributes are subsequently filtered through a global prototype alignment learning method, resulting in an instancedependent soft prefix. lastly, we employ a straightforward siamese representation learning approach for each instance to ensure stability throughout the training process. this methodical multi-step approach strives to reduce ambiguity and enhance the effectiveness of prefix-tuning in the realm of natural language processing. to comprehensively validate the efficacy of ccprefix, we conduct extensive experiments on three many-class classification tasks in both fully supervised and few-shot settings, including relation classification, topic classification and entity typing. the experimental results suggest that our work presents a promising step forward in the field, demonstrating the substantial potential of ccprefix in handling complex classification tasks in natural language processing. ## methodology in this section, we will provide a detailed explanation of our approach, with its overall architecture illustrated shown in figure 2 . task definition. first of all, we provide the task definition about the classification problem in finetuning paradigm. the classification tasks can be denoted as @xmath0, y}, where x is the instance set, @xmath1, y 2 , . . . , y |r| } is the class set, and |r| is the number of classes. the first token of the input is [cls] which contains the special classification embedding. plms models take the hidden state h of the first token [cls] as the representation of the whole sequence. a simple softmax classifier is then added to the top of plms to predict the probability of class y c : figure 2: our proposed model, ccprefix. for easy comprehension, we zoom out contrastive prefix construction and contrastive attributes generation in section 2.2. the losses l cls , l s and l con are defined in equation ( 9 ), equation (8) and equation ( 5 ). the black line is the forward path for both training and inference, while the green line is the training path with supervised signal. ## related work prefix tuning in classification. the templates can be categorized into two groups, i.e., discrete prompt @xcite @xcite and continuous prefix @xcite . discrete prompts often manually designed for all training instances with task descriptions. @xcite leverage manual logic rules to combine label-related sub-prompts together. although it is a concrete manifestation of human's interpretation of the task, discrete prompts may not be the optimal solution. continuous prefixes @xcite , attached to instances, have proven useful but fail to fully capture the diversity of training instances. though it has shown its merits, the shared prefix has ignored the diversity of training instances and has no contribution to discriminating the label space. our work inspired by the human decision process, introduces an instance-dependent prefix, better addressing the discrimination of label space. ## conclusion in this paper, we propose a novel task-agnostic approach named ccprefix. we sequentially construct fact-counterfacutal pairs to extract the attributes from the sample. with a set of global prototypes, the valid contrastive attributes will be selected as the prefix. a simple siamese represeatation learning is employed to stable the training process. the experiment results verify the superiority of our model without extra data and human experts for manually designing prefix templates. while our approach proves flexible for a broad spectrum of tasks in nlp, adapting it to causal language models (clms) presents operational challenges. we are committed to this exploration, recognizing its potential impact. we're also extending our work to include contrastive methods in clms for relation extraction tasks, aiming to increase our method's applicability across various models and tasks. this exploration signifies our method's potential for further expansion and adaptation in the field.