A weakly supervised textual entailment approach to zero-shot text classification (2023)

Abstract: Zero-shot text classification is a widely studied task that deals with a lack of annotated data. The most common approach is to reformulate it as a textual entailment problem, enabling classification into unseen classes. This work explores an effective approach that trains on a weakly supervised dataset generated from traditional classification data. We empirically study the relation between the performance of the entailment task, which is used as a proxy, and the target zero-shot text classification task. Our findings reveal that there is no linear correlation between both tasks, to the extent that it can be detrimental to lengthen the fine-tuning process even when the model is still learning, and propose a straightforward method to stop training on time. As a proof of concept, we introduce a domain-specific zero-shot text classifier that was trained on Microsoft Academic Graph data. The model, called SCIroShot, achieves state-of-the-art performance in the scientific domain and competitive results in other areas. Both the model and evaluation benchmark are publicly available on HuggingFace and GitHub.

## introduction ever since the first bert @xcite and gpt @xcite models were introduced to the world, the transformer @xcite has become the dominant architecture in the natural language processing (nlp) field. as a consequence, in the years that followed, the pretrainthen-finetune paradigm @xcite has been widely adopted to progressively push the state-of-the-art in a wide variety of downstream tasks and languages @xcite . even though the current training regime is far from being environmentally friendly due to the computational cost of pre-training @xcite , transfer learning removes the need of having to train a new model from scratch for each application. however, the fine-tuning of models for every single task is expensive both in terms of time and money as it is always preceded by a labor-intensive data labelling process @xcite . in order to overcome this issue, as well as the fact that real world data can be scarce in many scenarios, the field has started to shift towards techniques that require smaller amounts of labelled examples or even none at all @xcite @xcite . in particular, in the scientific domain, the growing amount of publications in an ever-increasing number of fields makes the classification task very challenging for neural language models @xcite . the impossibility to predict the emergence of new fields of study and the high cost associated with the creation of new datasets (which often requires domain experts) generate a need for systems that are capable of adapting to new situations. furthermore, the complexity and technicality of scientific language makes general-domain models perform poorly in comparison to domain-specific ones @xcite . this work addresses these problems by training an entailment-based zero-shot classifier for scientific text. instead of using a general domain dataset such as the popular xnli @xcite or mnli @xcite , a textual entailment dataset of scientific documents was built from scratch in a weakly supervised manner. by training a vanilla model on the entailment task, it is then able to classify documents into unseen classes with a high degree of success. as a by-product of this, the study of the relation between the training and target tasks led to intriguing questions about the strengths and limitations of the entailment approach to zero-shot text classification (zstc). ## related work zero-shot learning is a widely studied problem in machine learning that consists in completing a task for which no training examples were provided. although the term became initially popular in computer vision @xcite , it soon made the leap into nlp with an early paper @xcite that presented a classifier capable of interpreting the explicit semantic analysis (esa) representations of documents and labels from a semantic point of view, using wikipedia as a source of world knowledge. this highlighted the importance of capturing the labels' semantics in their representations, unlike in conventional text classification where labels are mapped to meaningless indices. this became easier to capture with the arrival of word embeddings, and can be enhanced by simply replacing their name with a short description @xcite . in subsequent research, zero-shot tasks were occasionally tackled from another problem's perspective @xcite , which implies training a model on a task for which annotated data is available and performing inference on a different one. recognizing textual entailment (rte) @xcite is arguably the most versatile task because of its generality, which is why it is commonly used to model other downstream tasks @xcite . the most popular approach at the time of writing is to reformulate zstc as a textual entailment problem as proposed by @xcite , aiming to imitate the way humans would address this natural language understanding task. the underlying idea is that a model trained on the entailment task should be able to perform classification of unseen classes by computing the entailment score between the input text, which acts as a premise, and candidate labels conveniently converted into hypotheses. later research showed that the next sentence prediction (nsp) objective for sentence pair classification can also be used as a strong baseline for zstc, since competitive results where obtained using raw bert models that were not fine-tuned on any natural language inference (nli) data @xcite . even though entailment-based zeroshot text classifiers have shown to have certain limitations like a high instability or an excessive reliance on spurious lexical patterns, the current literature offers no better alternatives for a problem that is still far from being mastered by machines. ## creation of weakly supervised data this section describes the methodology followed to transform a text classification dataset into entailment data that could potentially be used to fine-tune a domain-specific (or not) zero-shot text classifier. this approach takes advantage of the fact that there are plenty of publicly accessible labelled examples for classification while there is not so much available for entailment tasks, most likely due to the difficulty of producing this type of data. in this work, a weakly supervised nli dataset was constructed using microsoft academic graph foot_0 (mag) data as a starting point. to do so, all labels were converted to natural language sentences that would serve as hypotheses. therefore, the generated training examples consist of pairs of sequences (premise and hypothesis) that are delimited by eos tokens, where the first part of the text contains the abstract section from a scientific publication (premise) and the second part is an artificially generated sentence that somehow embeds the class label of the scientific text (hypothesis). once the classification data has been turned into the entailment format, any model can be fine-tuned by predicting whether the premise of each input sequence entails the corresponding hypothesis or not. note however that this is rather an adaptation of the rte task, since the second sentence does not really "entail" the first in positive examples. it is basically like performing a text classification task where labels are not converted into numeric indices, ensuring that their semantic content is preserved. figure 1 illustrates how a zstc model would operate both during fine-tuning (left) and at inference time (right), providing a full picture of the methodology followed in this work. we basically adopt the approach introduced by @xcite but going one step further by fine-tuning on a weakly supervised domain-specific dataset instead of using already existing general-domain nli data. figure 1 : overview of the entailment approach to zstc. on the left hand side it can be seen how the model is fine-tuned on a two-class textual entailment task, by providing the input text as a premise and the label name embedded in a natural language sentence that represents the hypothesis. on the right, the trained model classifies documents into unseen classes by computing the entailment score between the input text and each candidate label. ## results this section assesses the performance of our zeroshot classifiers, which were trained on a weakly supervised entailment dataset of scientific text. in an effort to obtain a more complete picture, we perform both an in-domain and out-of-domain study. ## analysis 7.1 rte vs zstc in the scientific benchmark this section analyses the relation between the training task and the zstc task to assess the effectiveness of the early stopping technique presented in section 5.3. we evaluate every checkpoint in our zstc scientific benchmark in the same way that several models were evaluated in section 5.2.