IBM MNLP IE at CASE 2021 Task 2: NLI Reranking for Zero-Shot Text Classification (2021)

Abstract: Supervised models can achieve very high accuracy for fine-grained text classification. In practice, however, training data may be abundant for some types but scarce or even non-existent for others. We propose a hybrid architecture that uses as much labeled data as available for fine-tuning classification models, while also allowing for types with little (few-shot) or no (zero-shot) labeled data. In particular, we pair a supervised text classification model with a Natural Language Inference (NLI) reranking model. The NLI reranker uses a textual representation of target types that allows it to score the strength with which a type is implied by a text, without requiring training data for the types. Experiments show that the NLI model is very sensitive to the choice of textual representation, but can be effective for classifying unseen types. It can also improve classification accuracy for the known types of an already highly accurate supervised model.

## system architecture we experimented with many different combinations of supervised, few-shot, and zero-shot techniques and submitted multiple such combinations for each of the case task 2 subtasks. despite their differences, all submissions are built on the same cascaded architecture of a supervised neural classification model followed by a neural nli-based reranking model. the submissions differ on the exact combination of classification model and reranking model. for each sentence, the classification model produces a ranked list of predicted types with scores, but only for the types the model was trained on. if the score of the top-ranked predicted type is below threshold, or if the top-ranked predicted type is other, the top n predicted types p n are passed to the reranking model along with all unseen types u . the reranker independently scores each of the types in p n ∪ u . the highest scoring type is the final prediction for the sentence. for each of the known and unseen types p n ∪ u to be submitted to the reranker, we generate a textual representation, based only on the definition of the type (not labeled examples). see section 4.2.1 for details on how we generate textual representations. the nli reranking model scores each of these representations on the extent to which they are implied by the input text. figure 1 illustrates the general architecture. ## supervised sequence classification for seen types for the text classifier we used a standard transformer-based sequence classification model @xcite with a pre-trained language model. based on previous experience with such text classification systems, we chose roberta @xcite . specifically, we started with the roberta-large model from hugging face @xcite . ## nli for unseen types pretrained language models (plms) have proven to be very powerful for downstream nlp tasks when they are fine-tuned on data specific to the task. recently, the research community has begun to observe that plms fine-tuned on large amounts of data for complex end-to-end tasks can often be leveraged for new tasks without further fine-tuning. fine-tuning plms on complex tasks such as question answering (qa) and natural language inference (nli) infuses models with high-level knowledge useful for other tasks. by choosing an appropriate task representation, qa models and nli models can be used as "pre-tuned" models for fewshot @xcite or even zero-shot @xcite text classification. typically, an nli model takes two texts (sen-tence1 and sentence2) and predicts whether sen-tence1 implies sentence2, with a given confidence score. to re-purpose an nli model for zero-shot text classification, sentence1 is the text to be classified and sentence2 is some textual representation of a type. the classification score for each type is the nli score, which represents the extent to which the textual representation of the type is implied by sentence1. determining implication is not just based on surface lexical overlap between the sentences. in training, the models learn encodings for both sentences, supervised by a large corpus of hand-labeled textual entailment pairs (such as the 433k sentence pairs in the multi-genre repeval corpus @xcite ). for the current work, we explored using large, pre-tuned nli models for few-shot and zero-shot classification. we experimented with nli extensions to both bart @xcite and roberta @xcite language models. for both of these, large models fine-tuned for the nli task are available from hugging face: bart-large-mnli and roberta-large-mnli. ## challenge submissions and results official task results appear in we now turn to a more detailed discussion of our submissions and more detailed scores. combining classifiers and rerankers, we arrived at eleven system configurations for submission to the challenge evaluation. in every case, the classification model for the known 25 acled event types was supervised. for the first three submissions of subtask 2, classification of the three unseen types was few-shot, trained on our small corpus (see section 3.1). for the fourth subtask 2 submission, we used the output of submission 1.3 as the "classifier". since submission 1.3 was tuned on data for the 25 known types only, submission 2.4 is zero-shot for the three un-seen types. no training data was used for the two new unseen types of subtask 3, so those types were always introduced during the reranking stage only (no classification). these were always zero-shot. reranking of the 25 original types was supervised when using the roberta nli model finetuned on the acled data (rob-mnli-ft24 in the table. when using roberta-large-mnli off-the-shelf, reranking the 25 was zero-shot. for the 3 unseen types of subtask 2 (and subtask 3), only submission 2.3 reranked using the nli model fine-tuned on the small amount of data for these types, and is considered few-shot. otherwise, all reranking of the 3 unseen types and the 2 unseen types of subtask 3 were zero-shot. ## observations and discussion the best performing configuration for subtask 2 overall was supervised classification for all 28 types with untuned (zero-shot) reranking. in particular, the zero-shot reranking (submission 2.2) outperformed the reranker tuned on the three unseen types. this runs counter to what we saw in the development experiments. the configuration that was most successful on the original 25 types was the one whose classifier was the 1.3 submission (which had a fine-tuned reranker). isolating the scores on the three unseen types versus the 25 known types shows strong performance in the fewshot case, but significantly weaker performance with zero-shot (2.4). for subtask 3, submissions 3.1 and 3.3 produced identical predictions (not just scores), as did submissions 3.2 and 3.4. the configurations themselves are not equivalent, with the input to the 3.3 and 3.4 rerankers having already been reranked by the 1.3 and 2.2 rerankers. interestingly, the configurations built on zero-shot rerankers only performed best, again suggesting nli models can be used without fine-tuning for reranking classification for both known and unseen types. performance on the two unseen types of subtask 3 (zero-shot) is significantly weaker than the zero-shot scenario of subtask 2 (2.4). it is possible that the two new types are inherently more difficult to recognize. but we suspect that tweaks to the textual representations for these two types might improve performance. given the extreme differences that different representations produce (section 4.1), we expect that more carefully chosen representations would help. ## conclusion the case 2021 task 2 challenge accurately simulates a realistic, fine-grained, text classification scenario in which many types in the type inventory have abundant labeled data, some types are recently new and may have a small amount of labeled data, and some types are completely new and have no labeled data. within these constraints, we proposed a hybrid system that combines supervised classification with nli-based reranking that can be used in supervised, few-shot, and zero-shot settings. our results show strong performance on known types with weaker results on unseen types. nevertheless, the experiments for this challenge have produced some interesting conclusions. first, we confirm that nli models are useful for zeroshot text classification, but only when distinguishing between a small number of target types. second, even in a fully supervised scenario, where ample training data can produce classification models with extremely high accuracy, untuned nli-based reranking can improve classification performance on known types. third, the choice of textual representation to transform a classification problem into one amenable to an untuned nli model greatly affects performance. in future work we hope to explore more rigorously what makes a good representation for nli-based zero-shot text classification, and how to generate these representations more automatically.