Unsupervised Natural Language Inference Using PHL Triplet Generation (2022)

Abstract: Transformer-based models achieve impressive performance on numerous Natural Language Inference (NLI) benchmarks when trained on respective training datasets. However, in certain cases, training samples may not be available or collecting them could be time-consuming and resource-intensive. In this work, we address the above challenge and present an explorative study on unsupervised NLI, a paradigm in which no human-annotated training samples are available. We investigate it under three settings: PH, P, and NPH that differ in the extent of unlabeled data available for learning. As a solution, we propose a procedural data generation approach that leverages a set of sentence transformations to collect PHL (Premise, Hypothesis, Label) triplets for training NLI models, bypassing the need for human-annotated training data. Comprehensive experiments with several NLI datasets show that the proposed approach results in accuracies of up to 66.75%, 65.9%, 65.39% in PH, P, and NPH settings respectively, outperforming all existing unsupervised baselines. Furthermore, fine-tuning our model with as little as ~0.1% of the human-annotated training dataset (500 instances) leads to 12.2% higher accuracy than the model trained from scratch on the same 500 instances. Supported by this superior performance, we conclude with a recommendation for collecting high-quality task-specific data.

## introduction natural language inference (nli) is the task of determining whether a "hypothesis" is true (entailment), false (contradiction), or undetermined (neutral) given a "premise". state-of-the-art models have matched human performance on several nli benchmarks, such as snli @xcite , multi-nli @xcite , and dialogue nli @xcite . this high performance can be partially attributed to the availability of large training datasets; snli (570k), multi-nli (392k), and dialogue-nli (310k). for new domains, collecting such training data is time-consuming and can require significant resources. what if no training data was available at all? in this work, we address the above question and explore unsupervised nli, a paradigm in which no human-annotated training data is provided for learning the task. we study three different unsupervised settings: ph, p, and nph that differ in the extent of unlabeled data available for learning. in ph-setting, unlabeled premise-hypothesis pairs are available i.e. data without ground-truth labels. in p-setting, only a set of premises are available i.e. unlabeled partial inputs. the third setting nph does not provide access to any training dataset, and thus it is the hardest among the three unsupervised settings considered in this work. we propose to solve these unsupervised settings using a procedural data generation approach. given a sentence, our approach treats it as a premise (p) figure 2 : comparing supervised nli with our three unsupervised settings. for unsupervised settings, we procedurally generate phl triplets to train the nli model. in nph setting, a premise pool is collected from raw text corpora such as wikipedia and then used for generating phl triplets. in p setting, we directly apply these transformations on the available premises. in ph setting, we leverage the p-setting model to pseudo-label and filter the provided unlabeled ph pairs and then train the nli model using this pseudo-labeled dataset. and generates multiple hypotheses (h) corresponding to each label (@xmath0, contradiction, and neutral) using a set of sentence transformations (refer to figure 1 ). this results in creation of premise-hypothesis-label (phl) triplets that can be used for training the nli model. in the p and ph settings, we directly apply our sentence transformations over the available premises to generate phl triplets. however, in the nph setting, premises are not available. we tackle this challenge by incorporating a premise generation step that extracts sentences from various raw text corpora such as wikipedia and short stories. we use these extracted sentences as premises to generate phl triplets. in figure 2 , we compare the four settings (one supervised and three unsupervised) and show our approach to develop an nli model for each setting. to evaluate the efficacy of the proposed approach, we conduct comprehensive experiments with several nli datasets. we show that our approach results in accuracies of 66.75%, 65.9%, and 65.39% on snli dataset in ph, p, and nph settings respectively, outperforming all existing unsupervised methods by ∼13%. we also conduct experiments in low-data regimes where a few human-annotated labeled instances are provided and show that further fine-tuning our models with these instances consistently achieves higher performance than the models fine-tuned from scratch. for example, with just 500 labeled instances, our models achieve 8.4% and 10.4% higher accuracy on snli and mnli datasets respectively. finally, we show that fine-tuning with 'adversarial' instances instead of randomly selected human-annotated instances further improves the performance of our models; it leads to 12.2% and 10.41% higher accuracy on snli and mnli respectively. in summary, our contributions are as follows: we release the implementation foot_0 of our procedural data generation approach and hope that our work will encourage research in developing techniques that reduce reliance on expensive human-annotated data for training task-specific models. unsupervised question-answering: the unsupervised paradigm where no human-annotated training data is provided for learning has mostly been explored for the question answering (qa) task in nlp. the prominent approach involves synthesizing qa pairs and training a model on the synthetically generated data. lewis et al. (2019); dhingra et al. (2018); fabbri et al. (2020) propose a template-based approach, while puri et al. (2020) leverage generative models such as gpt-2 (radford et al., 2019) to synthesize qa pairs. banerjee and baral (2020) create synthetic graphs for commonsense knowledge and propose knowledge triplet learning. wang et al. (2021) leverage fewshot inference capability of gpt-3 (brown et al., 2020) to synthesize training data for superglue (wang et al., 2019) tasks. for visual question answering, gokhale et al. (2020) use templatebased data augmentation methods for negation, conjunction, and banerjee et al. (2021) utilize image captions to generate training data. gokhale et al. (2021) use linguistic transformations in a distributed robust optimization setting for visionand-language inference models. unsupervised nli: in nli, cui et al. (2020) propose a multimodal aligned contrastive decoupled learning method (macd) and train a bert-based text encoder. they assign a label (e, c, n) based on the cosine similarity between representations of premise and hypothesis learned by their text encoder. our approach differs from macd as we leverage a procedural data generation step based on a set of sentence transformations and do not leverage data from other modalities. we use macd as one of the baselines in our experiments. ## unsupervised nli in nli, a premise-hypothesis pair (p, h) is provided as input and the system needs to determine the relationship @xmath1, contradiction, neutral} between p and h. in the supervised setting, a labeled dataset d @xmath2, h i ), l i } m @xmath3. however in the unsupervised setting, labels l i are not available, thus posing a significant challenge for training nli systems. along with this standard unsupervised setting (referred to as ph), we consider two novel unsupervised settings (p and nph) that differ in the extent of unlabeled data available for learning: ph-setting: it corresponds to the standard unsupervised setting where an unlabeled dataset of ph pairs ({(p i , h i )} m @xmath4. ## phl triplet generation to compensate for the absence of labeled training data, we leverage a set of sentence transformations and procedurally generate phl triplets that can be used for training the nli model. in p and ph settings, we apply these transformations on the provided premise sentences. in the nph setting where premises are not provided, we extract sentences from various raw text corpora and apply these transformations on them to generate phl triplets. ## training nli model in this section, we describe our approach to develop nli models for each unsupervised setting. ## conclusion and discussion we explored three different settings in unsupervised nli and proposed a procedural data generation approach that outperformed the existing unsupervised methods by ∼13%. then, we showed that fine-tuning our models with a few human-authored instances leads to a considerable improvement in performance. we also experimented using adversarial instances for this fine-tuning step instead of randomly selected instances and showed that it further improves the performance. specifically, in presence of just 500 adversarial instances, the proposed method achieved 70.85% accuracy on snli, 12.2% higher than the model trained from scratch on the same 500 instances. this improvement in performance suggests possibility of an alternative data collection strategy that not only results in high-quality data instances but is also resource efficient. using a model-in-the-loop technique has been shown to be effective for adversarial data collection @xcite @xcite @xcite . in these techniques, a model is first trained on a large dataset and then humans are instructed to create adversarial samples that fool the model into making incorrect predictions. thus, requiring the crowd-sourcing effort twice. however, in our method, a dataset designer can develop a set of simple functions (or transformations) to procedurally generate training data for the model and can directly instruct humans to create adversarial samples to fool the trained model. this is resource efficient and allows dataset designers to control the quality of their dataset.