Issues with Entailment-based Zero-shot Text Classification (2021)

Abstract: The general format of natural language inference (NLI) makes it tempting to be used for zero-shot text classification by casting any target label into a sentence of hypothesis and verifying whether or not it could be entailed by the input, aiming at generic classification applicable on any specified label space. In this opinion piece, we point out a few overlooked issues that are yet to be discussed in this line of work. We observe huge variance across different classification datasets amongst standard BERT-based NLI models and surprisingly find that pre-trained BERT without any fine-tuning can yield competitive performance against BERT fine-tuned for NLI. With the concern that these models heavily rely on spurious lexical patterns for prediction, we also experiment with preliminary approaches for more robust NLI, but the results are in general negative. Our observations reveal implicit but challenging difficulties in entailment-based zero-shot text classification.

## introduction natural language inference @xcite , also known as recognizing textual entailment (rte, @xcite , is normally formatted as the task of determining whether or not a premise sentence semantically entails a hypothesis sentence. the generality of the task format has aroused some recent studies to apply nli models for various downstream applications @xcite , and more recently text classification @xcite , making them generally-applicable solutions along with all those similar attempts to build a universal framework for various nlp tasks @xcite ). text classification is then reduced to textual entailment by setting * work during internship at microsoft research asia. the input sentence as the premise and simultaneously casting the candidate label into a hypothesis sentence using pre-defined templates or lexical definitions from wordnet. once we have any pretrained nli models at hand, zero-shot text classification under any specified label space is enabled for free without the need to collect annotated data. with contextualized representation based on pretrained language models such as bert @xcite , nli performance has been drastically improved. promising empirical results have been shown on various text classification benchmarks that vary across topic classification, emotion classification, and situation classification, outperforming earlier standard approaches @xcite or simple scoring schemes derived from distributional similarity @xcite . however, such generality is conceptually contradictory with the specificity of text classification in many practical scenarios. in this opinion piece, we conduct extended analysis on the recent attempts @xcite and point out some implicit issues under entailment-based zero-shot text classification that are overlooked in this line of work. we experiment with additional classification datasets and observe huge variance across them amongst standard bert-based nli models. more surprisingly, we find that raw bert models without fine-tuning can sometimes yield more competitive results. we also experiment with preliminary approaches for improving the robustness of nli models, but only to find negative results in general. our observations reveal implicit but massive difficulties in building a successful general-purpose zero-shot text classifier based on text entailment models. ## our investigation and implied issues we attempt at re-examining the earlier study @xcite with extended analysis to help estab-lish a better understanding of zero-shot text classification based on textual entailment. our focus is to check how well the models pre-trained for nli could generalize to the prediction of unseen categories, which is the major target of zero-shot classification. we did not study the setting that test set also include labels seen in training, commonly phrased as generalized zero-shot learning @xcite and referred to as the label-partiallyunseen setting by @xcite . that setting strongly assumes that a bunch of in-domain data for a number of classes are available already. foot_0 ## conclusion and discussion we investigate entailment-based zero-shot text classification further with extended analysis, uncovering the following overlooked issues: our observations reveal implicit but massive difficulties in building a usable zero-shot text classifier based on text entailment models. given the difficulty of nli data collection that aims at out-ofdomain generalization or transfer learning (bowman et al., 2020), we question the feasibility of this setup in the current progress of language technology. before significant progress in language understanding and reasoning, it seems more promising to consider alternative schemes built on explicit external knowledge @xcite @xcite or more crafted usage of pre-trained models that hopefully have captured more comprehensive semantic coverage and better compositionality from large corpora or grounded texts @xcite @xcite . this study also implies the huge difficulty for benchmarking zero-shot text classification without any further restriction on the task setting. the three datasets used by @xcite were originally intended for diverse coverage but are not sufficient to draw consistent conclusions as we have shown. we suggest future studies on zero-shot text classification either conduct experiments over even more diverse classification scenarios to verify any claimed generality, or directly focus on more specific task settings and verify claims within a smaller but clearer scope such as zero-shot intent classification or zero-shot situation typing for more reliable results with less instability, and perhaps based on more carefully curated data @xcite .