A synthetic data approach for domain generalization of NLI models (2024)

Abstract: Natural Language Inference (NLI) remains an important benchmark task for LLMs. NLI datasets are a springboard for transfer learning to other semantic tasks, and NLI models are standard tools for identifying the faithfulness of model-generated text. There are several large scale NLI datasets today, and models have improved greatly by hill-climbing on these collections. Yet their realistic performance on out-of-distribution/domain data is less well-understood. We explore the opportunity for synthetic high-quality datasets to adapt NLI models for zero-shot use in downstream applications across new and unseen text domains. We demonstrate a new approach for generating NLI data in diverse domains and lengths, so far not covered by existing training sets. The resulting examples have meaningful premises, the hypotheses are formed in creative ways rather than simple edits to a few premise tokens, and the labels have high accuracy. We show that models trained on this data (685K synthetic examples) have the best generalization to completely new downstream test settings. On the TRUE benchmark, a T5-small model trained with our data improves around 7% on average compared to training on the best alternative dataset. The improvements are more pronounced for smaller models, while still meaningful on a T5 XXL model. We also demonstrate gains on test sets when in-domain training data is augmented with our domain-general synthetic data.

## introduction over the past decade, nli tasks have been critical for benchmarking the representation strengths of our language models. today, the accuracy on the oft-reported multi-nli @xcite (mnli) dataset has reached above 92% 1 and supersedes human-level performance @xcite @xmath0: this book does a great job of putting all the different approaches under one roof, so that you can see what other researchers are doing and how they do it. h: the book covers different research approaches in a single place so you can compare them. l: entailment @xmath1: til the difference between "literally" and "figuratively". it was so easy to learn, i literally did a backflip. h: i did not bother learning the difference between "literally" and "figuratively". l: contradiction @xmath2: once upon a time, there was a very special young lady named cinderella. her stepmother and stepsisters were very mean to her. but she continued to be kind and helpful. h: cinderella was very kind to everyone. l: neutral mnli was a first effort to create a multi-domain nli training dataset with examples from 5 genres spanning fiction, formal texts and conversations, and with single-sentence premise/hypothesis texts. today, the use of models trained from such datasets has expanded well past routine benchmarking into a variety of practical tasks. in downstream problems involving web-scale llms, such as fact-checking of social media text, the domains and texts are clearly more diverse. yet, as a field, we have not fully explored the distribution-general abilities of 1 2212 our models for downstream semantic tasks beyond nli itself. there are many other nli training sets: anli @xcite for harder reasoning going beyond stylistic elements, and wanli @xcite that generates synthetic examples through worker and ai collaboration and which replicate complex reasoning patterns. to complement these efforts, we explore the opportunity for synthetic high-quality datasets to adapt nli models for zero-shot use in downstream applications across new and unseen text domains. our generation covers nearly 40 realistic and distinct domains, ranging from reviews, social media comments, to legal texts, also with varying lengths of the premise text. our technique employs a chain of llm tasks tuned to generate high quality, creative premise-hypothesis pairs together with a 3-way nli label (entailment, contradiction, or neutral @xcite ). a first step generates domain names, the second produces premises of different lengths in these domains, and the final llm call produces hypotheses and labels conditioned on each premise. we demonstrate how this approaches creates data with a balanced distribution of domains, labels, and premise lengths. we fine-tune nli models on this synthetic data corpus, and present their accuracy on the true factual consistency benchmark @xcite , consisting of 11 tasks unseen by our data and other training sets. we show that our general data-trained models obtain state-of-the-art nli performance and single-handedly outperform models trained on mnli, anli, or wanli with around 7% improvement over the best alternative for t5 @xcite small models. the gap is lower but still around 2% for t5 xxl model size. our main contribution is thus that the general synthetic data approach improves the generalization power of nli models, especially when small models and fast inference is key. we further show that, while in-distribution performance is hard to beat for tasks with in-distribution training data, our synthetic data can still improve in-distribution performance when used to augment the training data for models with sufficient capacity. ## related work we describe various ways of creating nli data, and why models should generalize beyond the data. ## synthesizing a general-nli dataset we aim to generate nli examples in different domains, and with premises of varied lengths. 3 generating synthetic examples for nli is in fact a challenging problem. the goal is to produce a pair of texts, which exemplify the reasoning behind different nli labels. but for the data to be useful, these texts must have creative content and language, 3 longer hypotheses are not of interest typically. a hypothesis is entailed if hypothesis is true given the premise. long hypotheses are less likely to contain precise entailment and contradiction relations, with some exceptions such as summaries. and require reasoning. prior synthetic data generation approaches @xcite generate the premise and hypothesis sentences as a single sequence, followed by annotating the label using a teacher model or human raters. instead, to achieve maximum control over multiple dimensions: genre, length, and different nli labels, we generate this dataset in two steps: (i) enumerate diverse domains and generate premises in those domains (section 3.1), (ii) generate hypotheses and labels given the premises (section 3.2). figure 1 provides an overall view into the llm tasks we use for generating our data. ## 2215 generate a text of a given size in the domain. given a sentence called premise, generate a related sentence called hypothesis. then generate a label explaining the relationship between the sentences. the options for the label are 'entailment', 'contradiction', and 'neutral'. entailment means that if the premise is true, the hypothesis is also true. contradiction means that if the premise is true, the hypothesis is false. neutral means that if the premise is true, we cannot say whether the hypothesis is true or false. premise: {premise} hypothesis: { ## experiments we explore the strengths of our general-dataset (gnli for brevity), by examining the model predictions on data unseen during training. we compare models trained on our data with those trained on other large nli training sets. we choose three such sources: the mnli dataset (392k training examples), anli (162k) with examples that are harder for mnli trained models, and wanli (102k), a dataset created by machinehuman collaboration. we note that all of these datasets are collected with a similar methodology, i.e., given a premise (and optionally a label), annotators (or llms) write a hypothesis. the final label is then manually assigned to the example (if not given as input). in addition, wanli and gnli have used mnli exemplars (few-shot or supervised learning) for data generation. so these datasets would have similar properties in theory. for all these sources, we trained t5 models (raffel et al., 2020) as a standard test bed, and explore models of different sizes: small (60m), large (770m), and xxl (11b). we trained on the respective training splits, and tune hyper-parameters on the corresponding validation sets. for wanli which does not contain a validation split, we used the mnli validation data. we also trained models on the combination of gnli and other datasets. for these combined models, we tuned 6 2217 hyper-parameters based on the classification accuracy on the development set of gnli. 10 we train two classifiers for each model size (e.g., t5 xxl) and training data (e.g., anli): a 3-way classifier with all the three labels, and a binary classifier. for the binary case, we convert each nli dataset into a binary dataset with entailment and non-entailment (neutral and contradiction) labels. we use the binary classifiers and 3-way classifiers for factual consistency evaluation and nli benchmarks, respectively. ## conclusion a decade of increasingly useful nli benchmarks and datasets have been instrumental in improving llms for various tasks. we have presented a new exploration of how the data distribution of each data source still impacts downstream performance on new examples. we proposed a synthetic data approach to mitigate these effects with examples balanced for domain, length and labels. we show that, by drawing on an llm's parametric knowledge of a broad range of domains, such synthetic data enables us to both train significantly more domain-general nli models, and to improve intrinsic nli model performance on in-domain test data by augmenting in-domain training data.