EDE ntail: An Entailment-based Few-shot Text Classification with Extensional Definition (2024)

Abstract: Few-shot text classification has seen significant advancements, particularly with entailment-based methods, which typically use either class labels or intensional definitions of class labels in hypotheses for label semantics expression. In this paper, we propose EDEntail, a method that employs extensional definition (EDef) of class labels in hypotheses, aiming to express the semantics of class labels more explicitly. To achieve the above goal, we develop an algorithm to gather and select extensional descriptive words of class labels and then order and format them into a sequence to form hypotheses. Our method has been evaluated and compared with state-of-the-art models on five classification datasets. The results demonstrate that our approach surpasses the supervised-learning methods and prompt-based methods under the few-shot setting, which underlines the potential of using an extensional definition of class labels for entailment-based few-shot text classification. Our code is available at https://github.com/MidiyaZhu/EDEntail.

## introduction entailment-based text classification formulates heterogeneous classification tasks into a unified textual entailment problem @xcite . unlike traditional classification models that often encode class labels into numerical vectors such as one-hot vectors without considering label semantics @xcite , the entailmentbased approaches express the semantics of class labels in the hypothesis and classify the input texts through semantic entailment matching between the input texts (premise) and the hypothesis. in hypothesis construction, class and subclass labels are commonly used as descriptive words for label semantics representation @xcite . the intensional definition of a class label * corresponding author by wordnet, which specifies the necessary and sufficient conditions of the class label @xcite , is also frequently utilized to provide label information @xcite . while such use of label semantic representations in hypothesis might be effective for some tasks, it may fall short in others because a single descriptive word or an intensional definition may not encapsulate all the semantic meanings within a label's domain, however, an effective hypothesis is expected to deduce various premises from the label domain. as shown in rows 1-4 in figure 1 , hypotheses constructed from the premises' class label fail to properly entail the given four premises in both language model and human cognition. besides, descriptive words that exhibit clear semantic entailment relationships in human cognition may exhibit dissimilarity in the word embedding space @xcite . the relationship learned from the same descriptive word might vary much in the word embedding space with different contexts (see section 4.7), hindering entailment feature learning. these limitations may weaken the entailment-based text classification performance, particularly in zero-shot or few-shot scenarios, when the test set lacks alignment with the hypothesis or when test samples fail to transfer the relation learned from the training samples. to address the above limitations in label semantics expression, we propose to construct hypotheses using extensional definition of class labels. an extensional definition, which gives the meaning of a term by listing all descriptive words that fall under this term foot_0 , provides explicit and diverse information of label semantics, facilitating semantic matching between different premises and a hypothesis. as illustrated in the last row of figure 1 , the label extensional definition-based hypothesis can subclass label class label subclass label 'a boring /negative /sad one' extensional definition 'a negative one' 'a sad one' 'a boring one' 'it expresses or consists of a negation or refusal or denial sentiment' intensional definition 'a film so tedious that it is impossible to care whether that boast is true or not' 'i feel so sorry that the picture failed to capture me' 'the film is strictly routine.' 'the movie makes absolutely no sense' entail all premises in both plm and human cognition. this not only broadens the semantics of class labels but also ensures computational efficiency by avoiding multi-time subclasses matching. at the linguistic level, descriptive words in the extensional definition can collaboratively reduce polysemy issues by refining semantic meanings with each other. at the machine-learning level, descriptive words in the extensional definition can boost embedding consistency among each other through the contextual learning capacity (see section 4.7). to implement the extensional definition-based hypothesis construction, we develop a generation method to ensure the descriptive words selected are representative and concise, considering that there could exist too many example words in the extensional definition of a class label and inclusion of all of them in hypotheses is impractical while randomly selection may result in insufficient coverage of the example words. our contributions are summarized below: 1. by analyzing current entailment-based methods, we identify that the semantic expression of labels within the existing hypothesis construction tends to be limited. this results in narrow coverage of label information and word embedding inconsistency in feature learning, hindering the performance of premise-hypothesis entailment. ## related work meta-task exhibits significant potential in zero-shot or few-shot text classification tasks. it can be classified into the generative method and the discriminative method @xcite . generative methods treat every task as a text-totext generation problem. prompt-based method as a generation method that treats the meta-task as a masked language modeling (mlm) problem @xcite . the mlm model predicts the masked token and then maps the predicted token to the label space through verbalizers. the prompt-based methods exhibit remarkable success in few-shot classification tasks @xcite . although knowledge can be incorporated into prompt verbalizer to enhance projection performance @xcite , the mlm model prediction may fall out of all possible associated candidates in verbalizer projection @xcite . discriminative methods like entailment-based method formulates meta-task under the framework of natural language inference, which aims to determine the relationship between the premise and the hypothesis as 'entailment', 'contradiction', or 'neutral' @xcite . recent studies in entailment-based methods include language model training @xcite @xcite , pseudo-label training @xcite , classifier training @xcite @xcite , and hypothesis engineering. the latter delves into the effective use of class label names @xcite or intensional definitions @xcite , or strategies for incorporating the above label information into ensemble models @xcite . one challenge here is the relationship learning between premise and hypothesis with extremely limited training sources @xcite , and another challenge is the construction of hypotheses suitable for all tasks @xcite . enhancing classification performance and stability under limited training data is a focal research point @xcite . in this paper, we propose a new method for hypothesis construction, aiming to address the aforementioned challenges in entailment-based text classification. ## proposed methods in this section, we introduce our proposed method, edentail, as depicted in figure 2 . this method comprises three key modules: extensional definition (edef) generation by gathering and selecting relevant and representative descriptive words; hypothesis construction by ordering and formatting words of extensional definition; and entailment reformulation for few-shot text classification. ## experiments in this section, we examine edentail's capabilities from the following perspectives: (1) the classification performance compared to other advanced classification models (section 4.3); (2) the classification performance compared with other entailmentbased models (section 4.4); (3) the efficiency of utilizing limited training samples (section 4.5); (4) the performance under zero-shot setting (section 4.6); (5) the effect of extensional definition on word embedding consistency (section 4.7); and (6) the classification performance compared with large language models (section 4.8). ## limitations as discussed in section 4.4, an entailment-based approach is not immune to potential biases arising from the uneven distribution of entailment and contradiction samples in multi-class test sets. to address this issue, we have implemented careful measures in our few-shot training method to achieve a balanced representation of both entailment and contradiction samples by ensuring that each label's extensional definition is represented as a contradiction sample at least once in train and valid set. however, further research is required to address the issue of sample number bias between entailment and contradiction in multi-class tasks.