Conformal Predictor for Improving Zero-Shot Text Classification Efficiency (2022)

Abstract: Pre-trained language models (PLMs) have been shown effective for zero-shot (0shot) text classification. 0shot models based on natural language inference (NLI) and next sentence prediction (NSP) employ cross-encoder architecture and infer by making a forward pass through the model for each label-text pair separately. This increases the computational cost to make inferences linearly in the number of labels. In this work, we improve the efficiency of such cross-encoder-based 0shot models by restricting the number of likely labels using another fast base classifier-based conformal predictor (CP) calibrated on samples labeled by the 0shot model. Since a CP generates prediction sets with coverage guarantees, it reduces the number of target labels without excluding the most probable label based on the 0shot model. We experiment with three intent and two topic classification datasets. With a suitable CP for each dataset, we reduce the average inference time for NLI- and NSP-based models by 25.6% and 22.2% respectively, without dropping performance below the predefined error rate of 1%.

## introduction zero-shot (0shot) text classification is an important nlp problem with many real-world applications. the earliest approaches for 0shot text classification use a similarity score between text and labels mapped to common embedding space @xcite @xcite @xcite . these models calculate text and label embeddings independently and make only one forward pass over the text resulting in a minimal increase in the computation. later approaches explicitly incorporate label information when processing the text, e.g., @xcite uses generative modeling and generates text given label embedding, and rios and kavuluru (2018) uses † work was done at salesforce ai research. label embedding based attention over text, both requiring multiple passes over the text and increasing the computational cost. most recently, nli- @xcite @xcite and nsp- @xcite based 0shot text classification formulations have been proposed. nli and nsp make inferences by defining a representative hypothesis sentence for each label and producing a score corresponding to every pair of input text and hypothesis. to compute the score, they employ a cross-encoder architecture that is full self-attention over the text and hypothesis sentences, which requires recomputing the encoding for text and each hypothesis separately. it increases the computational cost to make inferences linearly in the number of target labels. nli and nsp use large transformer-based plms @xcite @xcite and outperform previous non-transformerbased models by a large margin. however, the size of plms and the number of target labels drastically reduce the prediction efficiency, increasing the computation and inference time, and may significantly increase the carbon footprint of making predictions @xcite @xcite . in this work, we focus on the correlation between the number of labels and prediction efficiency and propose to use a conformal predictor (cp) @xcite to filter out unlikely labels from the target. conformal prediction provides a model-agnostic framework to generate a label set, instead of a single label prediction, within a pre-defined error rate. consequently, we use a cp, with a small error rate we select, based on another fast base classifier to generate candidate target labels. candidate labels are then used with the larger nli/nsp-based 0shot models to make the final prediction. we experiment with three intent classification (snips @xcite , atis @xcite and hwu64 @xcite ) and two topic classification (ag's news and yahoo! answers @xcite ) datasets using 0shot models based on a moderately sized bart-large (nli) @xcite and a small bert-base (nsp) plm. we use four different base classifiers, each with different computational complexity, and a small error rate of 1%. by using the best cp for each dataset, we reduce the average computational time by 25.6% (22.2%) and the average number of labels by 41.09% (43.38%) for nli-(nsp-) based models. ## methodology we improve the efficiency of nli/nsp models by restricting the number of target labels with a conformal predictor (cp). using a fast but weak base classifier-based cp, we produce the label set that removes some of the target classes for the 0shot model without reducing the coverage beyond a predefined error rate. ## experiments we evaluate our cp-based framework on the test set of intent (snips, atis and hwu64) and topic (ag's news and yahoo! answers) classification datasets. snips, atis, hwu64, ag's news and yahoo! answers have 7, 17, 64, 4 and 10 target labels, respectively. we use "facebook/bart-large-nli" and "bert-base-uncased" models from the hugging face hub @xcite as our 0shot nli and nsp baselines. our experimental setup is described in appendix ( §a). in fig. 1 and 2, we plot the empirical coverage and the average prediction set (aps) size of four base classifiers on atis, hwu64 and yahoo! an-swers datasets. coverage defines the proportion of samples for which the predicted set contains the 0shot model-predicted label. aps size equals the average number of labels in the set predicted by the cp base classifier. in ## discussion and future work in this work, we show that cp-based target label filtering improves the efficiency of nli-and nspbased zero-shot text classification models. our cp framework is generalizable to many formula-tions of zero-or few-shot models. for instance, prompt-based models that auto-regressively generate verbalizers @xcite are very slow, as the number of forward passes increases with the number of labels and verbalizer's tokens @xcite . our conformal prediction framework can be directly used there to reduce the number of labels and improve efficiency. additionally, the proposed conformal prediction framework can be used to filter training examples when constructing prompts for in-context learning (brown et al., 2020), where we prime plm with a sequence of training examples. for a task with many target labels, a naively constructed prompt would require at least one training example for each label. however, with cp, we can limit the number of labels (and consequently the number of training examples) in the prompt, minimizing the computational cost. in the future, we will explore newer methods to build conformal predictors that can further reduce the average prediction set size and inference time, as well as boost the performance of a 0shot model. ## limitations the datasets utilized in this research contain texts in english and thus mainly represent the culture of the english-speaking populace. gender or age biases may also exist in the datasets, and pre-trained models may also exhibit these biases. in addition, though the likelihood is slim, additional biases may also be introduced by cp-based label filtering. we recommend that any subsequent usage of the proposed technique clearly states that the conformal predictor was used for label filtering. in addition, while results suggest that our framework is generally effective for different zero-shot tasks, it should not be utilized on any new task without thorough evaluation, including evaluating for ethical or social risks.