Discriminatively- T uned G enerative C lassifiers for R obust N atural L anguage I nference (2020)

Abstract: While discriminative neural network classifiers are generally preferred, recent work has shown advantages of generative classifiers in term of data efficiency and robustness. In this paper, we focus on natural language inference (NLI). We propose GenNLI, a generative classifier for NLI tasks, and empirically characterize its performance by comparing it to five baselines, including discriminative models and large-scale pretrained language representation models like BERT. We explore training objectives for discriminative fine-tuning of our generative classifiers, showing improvements over log loss fine-tuning from prior work (Lewis and Fan, 2019). In particular, we find strong results with a simple unbounded modification to log loss, which we call the “infinilog loss”. Our experiments show that GenNLI outperforms both discriminative and pretrained baselines across several challenging NLI experimental settings, including small training sets, imbalanced label distributions, and label noise.

## introduction natural language inference (nli) is the task of identifying the relationship between two fragments of text, called the premise and the hypothesis @xcite . the task was originally defined as binary classification, in which the labels are entailment (the premise implies the hypothesis) or not entailment. subsequent variations added a third contradiction label. most models for nli are trained and evaluated on standard benchmarks @xcite @xcite in a discriminative manner @xcite . these benchmarks typically have relatively clean, balanced, and abundant annotated data, and there is no distribution shift between the training and test sets. however, when data quality and conditions are not ideal, there is a substantial performance decrease for existing discriminative models, including both simple model architectures and more complex ones. prior work on document classification and question answering has shown that generative classifiers have advantages over their discriminative counterparts in non-ideal conditions @xcite @xcite . in this paper, we develop generative classifiers for nli. our model, which we call gennli, defines the conditional probability of the hypothesis given the premise and the label, parameterizing the distribution using a sequence-to-sequence model with attention @xcite and a copy mechanism @xcite . we explore training objectives for discriminative fine-tuning of our generative classifiers, comparing several classical discriminative criteria. we find that several losses, including hinge loss and softmax-margin, outperform log loss fine-tuning used in prior work @xcite while similarly retaining the advantages of generative classifiers. we also find strong results with a simple unbounded modification to log loss, which we call the "infinilog loss". our evaluation focuses on challenging experimental conditions: small training sets, imbalanced label distributions, and label noise. we empirically compare gennli with several discriminative baselines and large-scale pretrained language representation models @xcite @xcite on five standard datasets. gennli has better performance than discriminative classifiers under the small data setting. moreover, when limited to 100 instances per class, gennli consistently outperforms all bert-style pretrained models on four of the five datasets. these results are appealing especially in comparison with bertstyle pretrained baselines. large-scale pretrained language models have achieved state-of-the-art results on a wide range of nlp tasks, but they still require hundreds or even thousands of annotated examples to outperform gennli. gennli also outperforms discriminative classifiers when the training data shows severe label imbalance and when training labels are randomly corrupted. we additionally use gennli to generate hypotheses for given premises and labels. while the generations tend to have low diversity due to high lexical overlap with the premise, they are generally fluent and comport with the given labels, even in the small data setting. ## a generative classifier for nli each example in a natural language inference dataset consists of two natural language texts, known as the premise and the hypothesis, and a label indicating the relation between the two texts. formally, we denote an instance x (p) , x (h) , y as a tuple consisting of a premise t }, and a label @xmath0. most existing nli models are trained in a dis-criminative manner by maximizing the conditional log-likelihood of the label given the input, i.e., log p(y | x (p) , x (h) ). in this paper, we propose generative classifiers for nli that are trained instead to estimate the probability of the hypothesis given the premise and the label, i.e., p(x (h) | x (p) , y), typically by maximizing log-likelihood. we decompose this conditional probability using the chain rule, and our final training objective is to minimize the following negative log likelihood: at inference time, the prediction is made as follows: throughout all of the experiments in this paper, we assume a uniform label prior p(y), so p(y) will not affect the argmax in eq. ( 2 ) and can be omitted. ## conclusions and future work we proposed gennli, a discriminatively-finetuned generative classifier for nli tasks, and empirically characterized its performance by comparing it to discriminative models and pretrained models. we found several discriminative fine-tuning objectives to outperform log loss, including infinilog, a simple but effective choice. we conducted extensive experiments with gennli, showing its robustness across challenging empirical conditions. we also showed its ability to generate hypotheses given premises and particular labels. future work may explore generating of diverse sets of hypotheses for a given premise and label, with the goal of performing data augmentation. other future work will be to measure the performance of gennli on adversarial and similarly challenging nli datasets.