Learning to Infer from Unlabeled Data: A Semi-supervised Learning Approach for Robust Natural Language Inference (2022)

Abstract: Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) aims at predicting the relation between a pair of sentences (premise and hypothesis) as entailment, contradiction or semantic independence. Although deep learning models have shown promising performance for NLI in recent years, they rely on large scale expensive human-annotated datasets. Semi-supervised learning (SSL) is a popular technique for reducing the reliance on human annotation by leveraging unlabeled data for training. However, despite its substantial success on single sentence classification tasks where the challenge in making use of unlabeled data is to assign “good enough” pseudo-labels, for NLI tasks, the nature of unlabeled data is more complex: one of the sentences in the pair (usually the hypothesis) along with the class label are missing from the data and require human annotations, which makes SSL for NLI more challenging. In this paper, we propose a novel way to incorporate unlabeled data in SSL for NLI where we use a conditional language model, BART to generate the hypotheses for the unlabeled sentences (used as premises). Our experiments show that our SSL framework successfully exploits unlabeled data and substantially improves the performance of four NLI datasets in low-resource settings. We release our code here: https://github.com/msadat3/SSL_for_NLI

## introduction natural language inference (nli) or recognizing textual entailment (rte) is the task of predicting whether a hypothesis entails, contradicts or is neutral to a given premise. it is widely used as a benchmark for evaluating natural language understanding (nlu) which plays a key role in many natural language processing tasks such as text summarization, machine translation and sentiment analysis. in addition to serving as a benchmark for nlu, nli has aided in improving the performance in downstream tasks such as fake news detection @xcite and fact verification @xcite . in recent years, deep learning based approaches @xcite @xcite have shown promising performance for nli due to their superior ability to extract deep semantic features. however, despite their success, one of the key challenges of the deep learning based models is that they require large scale human annotated datasets to perform well. consequently, earlier nli datasets such as sick @xcite and rte @xcite , while being instrumental in the progress of nli research, are not suitable for training these models due to their small size. to address this, large scale datasets such as snli @xcite , mnli @xcite , and anli @xcite ) have been proposed. the general annotation protocol for these datasets involves curating the premises from a pre-existing source and then employing human crowdworkers to write the hypotheses and assign their class labels. given the large number of annotated samples needed for a dataset to be suitable for the deep learning models, their construction requires a significant amount of human effort. consequently, creating new nli datasets capturing unique linguistic properties of different domains/time/demography becomes cumbersome and in some cases impossible. therefore, it is necessary to reduce the reliance on manually annotated data in training deep learning models for nli. to this end, we seek to harness unlabeled data by adopting semi-supervised learning (ssl) for low-resource nli datasets which could potentially improve the performance without the necessity of any additional human effort. ssl is a widely used method for automatically assigning pseudo-labels to unlabeled data to incorporate them into model training @xcite @xcite . however, unlike single sentence i'm sure i shall be only too delighted to make myself useful, i responded. ## a new ssl framework for nli this section details our proposed ssl for nli framework which consists of two components: hypothesis generation and self-training. an overview of our framework can be seen in algorithm 1. ## datasets we use the following datasets for our experiments. sick @xcite ) sick is a dataset containing 4,500 sentence pairs in its training set. we use the 8k imageflickr data set foot_0 to extract unlabeled premises. this dataset was one of the sources for deriving sick. rte @xcite this dataset contains ≈ 2.5k examples and was created by combining the rte1 @xcite , rte2 @xcite , rte3 @xcite and rte5 @xcite datasets. the premises for rte were extracted from wikipedia and news sources. therefore, we also extract sentences from wikipedia and the cnndm @xcite dataset to be used as unlabeled premises. since, the test set of rte is not publicly available, we use its development set as the test set and randomly sample a small subset from the training set to be used as the development set. mnli -6k @xcite to simulate a low-resource environment, we randomly sample 6,000 examples from the training set of mnli. from the rest of the training set, we select the premises which do not occur in the sampled 6,000 examples to be used as unlabeled data. similar to rte, we use the development set of mnli as the test set and sample a small subset of examples from the training set to be used as development set. snli -6k @xcite we created snli -6k in a similar fashion as mnli -6k. ## baselines given that we are the first to explore nli in a low-resource setting, there are no existing methods that we can use as baselines, which are specifically tailored for nli when human annotated data is limited. thus, most of our baselines are adopted from methods which have been successful for other nlp tasks in low-resource settings. specifically, we compare the performance of our ssl framework with three types of baselines. bert @xcite : this is a baseline in which a bert model is fine-tuned only on the available human-annotated data. ## main experiments & results our main experiments and results are described in this section. we run each of our self-training and baseline experiments three times and report the average and standard deviation of their macro f1 scores in dbst vs bert to understand the effectiveness of our ssl framework, we compare the performance of our dbst model with the baseline bert model. the results show that dbst substantially improves the performance over bert for all four datasets. for example, we can see an increase of 7.08% and 3.97% in macro f1 by dbst over bert on rte and mnli-6k mm , respectively. our qualitative analysis of the syntheticallylabeled data suggests that the bart models used in our framework are able to generate meaningful hypotheses for each premise (see dbst vs da baselines next, we evaluate whether our ssl framework can improve the performance over prior methods which handle lowresource scenarios without leveraging unlabeled data. to this end, we compare the performance of the dbst model with our da baselines. we can see in ## out-of-domain results to assess whether our ssl framework can improve the out-of-domain (ood) performance, we evaluate dbst + n and bert models trained on each dataset using test sets of other datasets. it should be noted that we do not perform any additional training of the models for evaluating the ood performance. we simply use the models already trained on labeled and/or unlabeled data on the training set of a particular dataset and test them on other datasets. the ood results are reported in ## analysis robustness analysis from our experiments and results, it is evident that our ssl framework is effective in improving both in-domain and out-ofdomain performances. however, it is not clear if ssl can train more robust nli models. therefore, we study the robustness of our models on nli stress test @xcite . the stress test was created based on the weaknesses of nli models in various aspects such as word overlap -models tend to predict a sample to be entailment if there is a large overlap in words even if they are unrelated, negation -presence of a negation word such as 'no' causes the model to predict the sample as contradiction etc. in total, 11 different tests are carried out which are divided into three parts: competence, distraction, and noise. we compare the stress test results for the baseline bert model and dbst + n model in the dbst + n models show better performance than the baseline bert model on both distraction and noise tests. therefore, our ssl framework reduces the vulnerability of the models in being distracted by shallow features such as word overlap, single bart in order to understand the necessity of using separate bart models for each class, we perform an experiment on mnli-6k where we employ a single bart model to generate the hypotheses for all classes in our dbst + n approach. to this end, we append the label at the end of each premise in the labeled training set to be used as the source text and use their hypotheses as the target text to fine-tune a bart model. similarly, to generate the hypotheses for different classes, we append the class labels at the end of the unlabeled premises before using them as the input to the fine-tuned bart model. a comparison between dbst + n and dbst + n sb on mnli-6k can be seen in ## conclusion & future work in this paper, we propose a novel method for leveraging unlabeled data for nli in low-resource settings to improve the performance without the necessity of additional human annotation effort. we develop an ssl framework based on this proposed method for nli which substantially improves both in-domain and out-of-domain performance of four nli datasets in low resource scenarios illustrating that our ssl framework considerably improves the generalization capability of the models by harnessing unlabeled data. our results indicate that we successfully address the key bottleneck in exploring ssl for nli, i.e., the unavailability of hypothesis for unlabeled premises. our future work will in-clude developing methods which can improve the robustness of the models further.