C on E ntail: An Entailment-based Framework for Universal Zero and Few Shot Classification with Supervised Contrastive Pretraining (2023)

Abstract: A universal classification model aims to generalize to diverse classification tasks in both zero and few shot settings. A promising way toward universal classification is to cast heterogeneous data formats into a dataset-agnostic “meta-task” (e.g., textual entailment, question answering) then pretrain a model on the combined meta dataset. The existing work is either pretrained on specific subsets of classification tasks, or pretrained on both classification and generation data but the model could not fulfill its potential in universality and reliability. These also leave a massive amount of annotated data under-exploited. To fill these gaps, we propose ConEntail, a new framework for universal zero and few shot classification with supervised contrastive pretraining. Our unified meta-task for classification is based on nested entailment. It can be interpreted as “Does sentence a entails [sentence b entails label c]”. This formulation enables us to make better use of 57 annotated classification datasets for supervised contrastive pretraining and universal evaluation. In this way, ConEntail helps the model (1) absorb knowledge from different datasets, and (2) gain consistent performance gain with more pretraining data. In experiments, we compare our model with discriminative and generative models pretrained on the same dataset. The results confirm that our framework effectively exploits existing annotated data and consistently outperforms baselines in both zero (9.4% average improvement) and few shot settings (3.5% average improvement). Our code is available in supplementary materials.

## introduction it has been a long-standing effort to solve various text classification tasks by training one universal model @xcite . with an ideal universal classification model, we can expect extreme generalization with few or zero annotation in new domains/tasks/datasets. to this end, researchers reformulate heterogeneous task definitions into a unified format of a meta-task in natural language @xcite . solving the meta-task is equivalent to solving the isolated tasks, thus the meta-task paves the way of supplementing unsupervised pretrained language models (plm) with additional supervised pretraining, to further absorb knowledge from heterogeneous labeled data. the success of universal classification models hinges on how well a strong plm understands natural language meta-task. the meta-task format depends on two underlying plm types: (a) discriminator uses encoder plms and treats all classification tasks as binary entailment classification problem @xcite @xcite . however, they only pretrain models on natural language inference datasets, whose knowledge is not comprehensive comparing all classification tasks @xcite . (b) generator uses encoder-decoder plms and treats all tasks as text generation problem @xcite @xcite @xcite @xcite . thus they are compatible with both classification tasks and generation tasks. however, the generator nature implies that the predicted texts may not match any possible labels, thus more likely to fail on classification tasks @xcite . based on our observations and experiments, we argue that the discriminators have more potential in universal classification, and propose a new discriminator framework, conentail, that can make better use of existing annotated datasets. concretely, we reformulate the unified meta-task as a nested entailment: "does sentence q entails [sentence p entails label h]". take fig. 1 as an example, the query "we had a great breakfast at the waffle shop!" entails the same label as the premise "i bought this for myself a short time ago and i love it. an excellent in a few-shot setting, the premise is an example sentence; in a zero-shot setting, the premise is a "null" placeholder. in experiments, we collect 56 classification datasets from crossfit @xcite , together with their templates, to formulate a large supervised pretraining dataset. we reproduce efl @xcite , unifew @xcite and crossfit @xcite in the same setting and control influences of plm supervised pretraining data, then conduct fair comparison with our proposed co-nentail. the experiments show that generators (unifew and crossfit) do not fit the classification task well and thus significantly under-perform the random guess in zero-shot evaluation; standard discriminators (efl) under-exploit supervised pretraining datasets and thus do not gain consistent improvement as pretraining data scale up, while conentail makes the best use of the supervised pretraining data and keep consistent performances. our model outperforms baselines in both zero (9.4% average improvement) and few shot settings (3.5% average improvement). our contributions are the following: supervised pretraining supervised pretraining originates from explicit multitask learning @xcite which combines different task knowledge into shared representations. @xcite found that supplementing plms with supervised pretraining between unsupervised pretraining and downstream finetuning can significantly boost the performance and few-shot generalization. the discriminator models including ufo-entail @xcite and efl @xcite are trained on mnli @xcite in a supervised fashion, but they do not combine different sources of datasets. furthermore, t0 @xcite and ext5 @xcite extends t5 (raffel et al., 2020) by using 107 and 171 datasets for supervised pretraining and conduct zero-shot evaluation. flex @xcite and crossfit @xcite extends the supervised pretraining evaluation to few-shot learning. the supervised pretraining strategies from these works vary in pretraining datasets and hyperparameters, but they mostly follow their underlying language model tasks, such as next sentence prediction or text generation. we argue that applying the unsupervised pretraining strategy to supervised pretraining is an underuse of the labeled data, and propose a supervised contrastive learning method on plms for better zero/few-shot generalization. contrastive learning for nlp contrastive learning aims to create embeddings such that similar examples are close while dissimilar examples are far away @xcite . while most works use self-supervised contrastive learning @xcite @xcite , only a few adopt supervised contrastive learning. clip @xcite uses labeled images and captions as supervision signal. sim-cse @xcite and sbert (reimers and gurevych, 2019) use labeled sentence pairs from nli to construct positive and negative examples. however, their contrastive data creations are limited to specific types of data, and thus can be hardly extended to universal classification. we reformulate all nlp classification tasks into a unified contrastive meta-task and use supervised contrastive loss @xcite to train on heterogeneous labeled data during supervised pretraining. 3 method ## results and analysis we design the following experiments to demonstrate and analyze the effectiveness of our method. first, we present the best scores of the compared models with or without supervised pretraining as our main result (section 5.1). then, we investigate the performance gain or loss of each model brought by the supervised pretraining (section 5.2). furthermore, we study the fine-grained impact of more labeled data in supervised pretraining or of more labeled data in support set (section 5.3). considering these results, we discuss the difference between discriminators and generators (section 5.4). finally, we show a case study of universal classification under a zero-shot scenario (section 5.5).