FL am E : Few-shot Learning from Natural Language Explanations (2023)

Abstract: Natural language explanations have the potential to provide rich information that in principle guides model reasoning. Yet, recent work by Lampinen et al. has shown limited utility of natural language explanations in improving classification. To effectively learn from explanations, we present FLamE, a two-stage few-shot learning framework that first generates explanations using GPT-3, and then fine-tunes a smaller model (e.g., RoBERTa) with generated explanations. Our experiments on natural language inference demonstrate effectiveness over strong baselines, increasing accuracy by 17.6% over GPT-3 Babbage and 5.7% over GPT-3 Davinci in e-SNLI.Despite improving classification performance, human evaluation surprisingly reveals that the majority of generated explanations does not adequately justify classification decisions. Additional analyses point to the important role of label-specific cues (e.g., “not know” for the neutral label) in generated explanations.

## introduction collecting and learning from natural language explanations has received increasing attention in the nlp community @xcite . the idea of learning from natural language explanations is especially appealing in few-shot learning because explanations can provide rich information about the task and guide model reasoning when there are limited supervision signals. although large-scale language models (llms) have demonstrated a remarkable capability in fewshot learning @xcite @xcite , the effect of learning from natural language explanations remains mixed. on the one hand, @xcite demonstrates impressive success with chain-of-thought prompting, especially in arithmetic reasoning. on the other hand, in a systematic evaluation of the effect of explanations on in-context learning, @xcite discover only a marginal improvement from explanations, even when experimenting with massive models (280b). it thus remains an open question how we can leverage llms to effectively learn from natural language explanations. we propose a two-stage framework (🔥flame) for few-shot learning from natural language explanations. fig. 1 gives a graphical overview of our approach. first, our framework leverages the ability of large-scale language models (e.g., gpt-3) to generate explanations. second, it uses explanationaware prompt-based classification where we can fine-tune a smaller model (e.g., roberta). the second step enables the model to tailor to the imperfect explanations from gpt-3 and also opens up opportunities to interpret and probe the model given its transparent internals. we show that flame outperforms strong baselines in natural language inference. compared to gpt-3 finetuned with explanations, flame achieves higher accuracy than babbage by 17.6% on e-snli and 6.9% on e-hans, and also outperforms davinci by 14.2% on e-snli and 14.3% on e-hans. in addition, flame outperforms the strongest baselines that do not use explanations by 5.7% on e-snli and 1.2% on e-hans. furthermore, we conduct an in-depth analysis to understand how our approach improves classification and reveal the important role of label-specific cues. we first show that the generated explanations do not perform valid inferences according to human evaluation. this result corroborates recent work on the characteristics of gpt-3 explanations: they read fluent but lack accurate reasoning @xcite . we also observe that gpt-3 explanations frequently include tokens that encode label information (e.g., "not know" for the neutral label). our two-staged framework uses a small classification model, enabling us to probe the behavior of our model with perturbed explanations. to inves- (1) explanation generation and (2) explanation-aware classification. we use distinct colors to represent labels, and use to indicate a generated explanation. in stage 1, flame generates an explanation for each label @xmath0. in stage 2, flame uses a prompt-based model to classify with the aid of explanations. specifically, for each label y and generated explanation , we measure the (unnormalized) probability f (y| ) of unmasking y from the prompt in the presence of , and the predicted label ŷ is the label associated with the maximum probability in the matrix. it is worth noting that our main experiments were done with the gpt-3's fine-tuning api due to our preliminary experiments and budget considerations. we later found that our performance improvement in e-snli is robust against gpt-3 in-context learning with davinci and babbage, but it is not against gpt-3 davinci in e-hans, likely due to the templated nature of e-hans. this discrepancy between in-context learning and fine-tuning with gpt-3 motivates future work to understand and control these black-box models. in summary, our contributions are: • we propose flame, a few-shot learning framework that effectively leverages natural language explanations to improve classification. ## learning from explanations our method (flame) consists of two stages: 1) explanation generation with gpt-3 and 2) explanation-aware classification with a smaller standalone model (fig. 1 ). deviating from the paradigm in literature of treating both processes as a joint optimization problem @xcite , the disentanglement of explanation generation from classification allows our methods to use the capability of large language models to generate fluent explanations from a handful of examples, while leaving classification to a downstream model, thereby enabling probing experiments and explicit control over the classification component. ## experimental setup in this section, we present our experimental setup and discuss important choices in implementation. we will release our code upon publication. ## results we demonstrate that our framework on learning from explanations is effective as it reliably outperforms baselines across datasets and conditions (4.1), and we analyze why and how explanations are useful in our framework (4.2, 4.3). ## gpt-3 in-context learning since openai reduced its api pricing, the authors decided to obtain in-context learning results for gpt-3 no-explanation. even if we increase gpt-3 model size to 175b (davinci), flame still outperforms in-context learning on e-snli (+8.1%). similar to babbage, fine-tuning provides better performance than incontext learning in e-snli. in contrast, gpt-3 davinci in-context learning performs better on e-hans, likely due to its templated nature. according to the induction heads hypothesis @xcite , in-context learning uses two kind of attention heads to copy and complete patterns. gpt-3 davinci may utilize this mechanism to achieve high performance on e-hans. the divergent behavior between fine-tuning and in-context learning requires additional investigation. it further motivates research on controlling these black-box models that are not easily accessible to the majority of researchers. ## related work we review additional related work in natural language explanations (nles), few-shot learning, and model distillation. ## conclusion we present 🔥flame, a two-stage framework that leverages the few-shot generation capability of gpt-3 and a relatively small model to effectively use the generated explanations with fallible reasoning. our approach outperforms strong baselines in natural language inference. we further show that while the generated explanations are invalid, they include useful label-specific cues. through a probing experiment, we prove that these label-specific cues are essential for model prediction. we believe that using a smaller model to leverage the outputs from large language models is a promising direction for future work. this approach has at least two advantages: 1) the small model can potentially handle the imperfect outputs from the large model; 2) the small model allows for efficient interpretation and probing of the final pipeline. future work may investigate removing the dependency on the large model altogether at test time.