Few-Shot Self-Rationalization with Natural Language Prompts (2022)

Abstract: Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB—a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.

## introduction models constrained to be more understandable to people are easier to troubleshoot and more useful in practice @xcite . for instance, constraining a model that answers the question "which linguist invented the lightbulb?" with "none" to also provide the reason-"thomas edison is the inventor of the lightbulb and he was not a linguist"makes the model easier to control and interact with @xcite . models that jointly predict task labels and generate free-text explanations for their predictions (as in the previous example) are known as self-rationalization models @xcite . their explanations are arguably more faithful and stable than post-hoc explanations since * equal contributions. we then study the impact of model size on fewshot self-rationalization to investigate whether the quality of generated explanations scales with the size as good as the accuracy of predicting task labels. to this end, we also evaluate gpt-3's @xcite self-rationalization behavior. our experiments show that explanation plausibility scored by human annotators (which range from 0-100) and end-task accuracy improve with increasing model size. specifically, the difference in plausibility scores between the base and 3b model ranges from [6.2, 24.8] (on average 14.8). the average plausibility across datasets is 43.4 (unifiedqa-3b) and 50.6 (gpt-3). while encouraging, our results show that there is still a large gap between model and human performance (25.7 for gpt-3), and we hope this work will help enable the research community to take on the few-shot selfrationalization challenge. our code for producing data splits, prompt construction, model training/evaluation, and human evaluation templates are publicly available. foot_1 ## feb benchmark there has been an explosion of interest in generating free-text explanations and in few-shot learning in the last 1-2 years. however, appropriate datasets and metrics for few-shot self-rationalization have not yet been established. we thus introduce the feb benchmark-a suite of existing englishlanguage datasets with human-authored free-text explanations and associated metrics for few-shot self-rationalization. we expect that feb will simplify future model comparison and lower barriers to entry for those interested in working on this task. ## prompting for self-rationalization we approach few-shot self-rationalization with prompt-based finetuning using natural language (nl) prompts. the key idea behind nl prompts is that a pretrained language model (lm) is already well-positioned to solve the end-task if we format finetuning end-task examples as similar as possible to the format used in the lm's pretraining. following that principle, in this section, we describe our prompting approach with t5 @xcite and comprehensively evaluate three distinct prompt types with feb. our results show that a unified question-answering (qa) prompt combined with a t5 variant that includes additional supervised multitask qa training (unifiedqa; @xcite performs the best overall across tasks, when compared to three different alternative prompts as described below. self-rationalization models @xcite are currently based on t5 for at least two reasons. first, t5 has been pretrained with many supervised tasks including classification and generation tasks, and self-rationalization involves both classification and generation. second, t5 is one of the largest open-sourced and widely studied pretrained models, and higher lm performance is correlated with larger model size @xcite . thus, all of our experiments are based on t5 (and the unifiedqa variant when evaluating prompts based on a qa format). in this section, all results are obtained with the base version of these models and in §4 we scale model size. when a lm is pretrained with masked language modeling @xcite only, an appropriate nl prompt is constructed by adding and infilling masked tokens @xcite . t5, however, has been pretrained with span infilling and a suite of supervised tasks whose instances were formatted in various ways. one of these supervised tasks includes squad 1.1 @xcite ) which allows us to experiment with prompts based on qa templates. as a result, we were able to design several different types of nl prompts for t5 consistent with different aspects of its pretraining: ## improving self-rationalization with increasing model size in §3, we discovered that a qa prompt combined with the base unifiedqa model version is as an effective combination for few-shot selfrationalization through prompt-based finetuning. in this section, we provide two additional evaluations to establish the first approach to few-shot self-rationalization. first, we assess how plausible the generated explanations are when evaluated by annotators on amazon mturk. details of how we conduct human evaluation of plausibility are given in §2. one hit contains 10 instances and we pay @xmath01,050, we do not do prompt search for gpt-3. we use the prompts shown in fig. 1 in appendix. a detailed description of evaluation metrics is given in §2. the dev set size (of each out of 60 dev sets) for gpt-3 is 18 instead of 350 (because of the api cost). ground-truth explanations are evaluated together with explanations generated by 4 models. therefore, for gold explanations, we report the average of 4 plausibility scores, std. errors, and κ values calculated with 4 mturk batches (corresponding to 4 models). ## related work few-shot self-rationalization a standard approach to creating explanations in the form of highlights is the select-then-predict method @xcite that does not use any human-author input highlights. on the other hand, a standard method for generating free-text explanations is to use human-written explanations @xcite @xcite . to the best of our knowledge, prior to submitting our work only two prior works have generated free-text explanations in a weakly-supervised way from the task prediction loss. @xcite approach commonsense qa in that fashion. @xcite propose a distant supervision approach to explaining a defeasible inference task. in this paper, we introduce the feb benchmark to unify the evaluation of few-shot selfrationalization and present the first approach and results on feb. concurrent to our work, yordanov et al. (2021) study self-rationalization transfer from a highresource task to a task with only a few humanauthored explanations. @xcite analyze explanations obtained by prompting gpt-3 multiple times to get multiple explanation candidates, and then filter these candidates using a model trained to predict acceptability of explanations. their prompt consists of a few examples with high-quality explanations written by the authors and a new instance together with its gold label. @xcite demonstrate end-task performance improvements attained by prompting the palm model @xcite to first generate an explanation behind its reasoning ("chain of thought") and then the task label. zelikman et al. (2022) extend this approach by using explanations generated in a few-shot manner to refine the same gpt-j (wang and komatsuzaki, 2021) model. ## conclusions we draw attention to the task of few-shot selfrationalization: predicting task labels and generating free-text explanations for the prediction using only a few human-written explanations. we present (i) the feb benchmark, (ii) the first prompting approach for feb established through a comprehensive search of natural language prompts, and (iii) results using models with a number of parameters ranging from 220m to 175b. our human evaluation results show that progress is possible on this task given that just scaling the model size increases both the plausibility of generated explanations and task accuracy by a very large margin. despite that, few-shot self-rationalization remains very challenging, with the plausibility of explanations generated by the best model being 27.7 points behind that of human-authored explanations. we hope that work presented in this paper spurs the community to work on this challenging problem to enable more intuitive interaction with nlp systems. feb task similar t5 pretraining tasks e-snli mnli @xcite classify the entailment relation between two sequences ecqa record @xcite answer a cloze-style query about a passage given entities in it comve copa @xcite select one of two sequences as the cause/effect of a premise sbic cola @xcite classify a sentence as acceptable or not