Investigating the Effect of Natural Language Explanations on Out-of-Distribution Generalization in Few-shot NLI (2021)

Abstract: Although neural models have shown strong performance in datasets such as SNLI, they lack the ability to generalize out-of-distribution (OOD). In this work, we formulate a few-shot learning setup and examine the effects of natural language explanations on OOD generalization. We leverage the templates in the HANS dataset and construct templated natural language explanations for each template. Although generated explanations show competitive BLEU scores against ground truth explanations, they fail to improve prediction performance. We further show that generated explanations often hallucinate information and miss key elements that indicate the label.

## introduction thanks to recent advances in pre-trained language models @xcite , the state-of-the-art accuracy for natural language inference (nli) can easily exceed 90% @xcite . however, these nli models show poor out-of-distribution (ood) generalization. for instance, @xcite create a templated dataset (hans) and find model performance to be about chance in this dataset. while recent studies try to tackle this robustness problem from the perspectives of both the dataset and the model @xcite @xcite , we investigate an extra dimension of information, natural language explanations. our work is motivated by the growing interest in explanations in the nlp community @xcite @xcite ): these explanations can potentially enable models to understand the reasoning strategy beyond spurious patterns. we focus on a few-shot learning setup because it is unrealistic to expect a large number of annotated ood examples. to introduce an ood setting with natural language explanations, we construct e-hans, a dataset with natural language explanations for each template in hans. by leveraging the templates in hans, we avoid the challenges in crowdsourcing natural language explanations @xcite and manually build an explanation dataset of high-quality. we use an explainthenpredict framework to learn with explanations. an explanation generation model outputs an explanation for each input example, and the generated explanation is fed into a classifier along with the input example. while bleu scores imply high quality of generated explanations, learning with explanations does not improve predictive performance either in-distribution or out-of-distribution. we show the generated explanations contain words in the true explanations, but they fail to reproduce important phrases and often hallucinate entities during generation. ## building natural language explanations for hans to investigate whether natural language explanations improve the robustness of natural language inference (nli), we build on two existing datasets: 1) hans, which introduces templates to generate ood examples for robust evaluation of nli models; 2) e-snli, which provides explanations for the stanford natural language inference (snli) dataset. our key contribution is to augment hans by building templated natural language explanations and studying the effect of these explanations on model robustness in a few-shot learning setup. our dataset is available at https://github.com/chicagohai/hansexplanations . ## results we first look at the quality of generated explanations using bleu. although the generated explana-tions match groundtruth explanations well based on bleu, they affect downstream classification negatively in our few-shot learning set-up. we further examine the generated explanations to understand why the predictive performance drops when adding natural language explanations. ## conclusion we construct a hans-based dataset with explanations. on this dataset, we find natural language explanations do not help few-shot nli to generate out-of-domain under an explainthenpredict framework. while the genearted explanations obtain high bleu scores, they do not learn information crucial for downstream classification. our generation model is close to the sota model, yet it still generates nonsensical explanations. better metrics for explanation evaluation and explanation generation models are key to success for learning with natural language explanations to be effective.