I 2 D 2: Inductive Knowledge Distillation with N euro L ogic and Self-Imitation (2023)

Abstract: Commonsense capabilities of pre-trained language models dramatically improve with scale, leading many to believe that scale is the only winning recipe. But is it? Here, we investigate an alternative that a priori seems impossible: can smaller language models (e.g., GPT-2) win over models that are orders of magnitude larger and better (e.g., GPT-3), if powered with novel commonsense distillation algorithms?The key intellectual challenge is to design a learning algorithm that achieve a competitive level of commonsense acquisition, without relying on the benefits of scale. In particular, we study generative models of commonsense knowledge, focusing on the task of generating generics, statements of commonsense facts about everyday concepts, e.g., birds can fly. We introduce I2D2, a novel commonsense distillation framework that loosely follows the Symbolic Knowledge Distillation of West et al. but breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model’s own enhanced commonsense acquisition capabilities. Empirical results suggest that scale is not the only way, as novel algorithms can be a promising alternative. Moreover, our study leads to a new corpus of generics, Gen-A-tomic, that is the largest and highest quality available to date.

## introduction language models (lms) become better with scale. however, even the largest lms continue to fail in unexpected ways due to their lack of commonsense @xcite . knowledge models -custom lms trained to generate knowledgeprovide on-demand access to task-specific knowledge to address this gap @xcite . ## the i2dframework i2d2 is a new framework for automatically generating generic statements using pretrained language models. our language model of choice is gpt-2 xl. however, any auto-regressive language model can be used within i2d2. 3 i2d2 generates generics in four stages. first, in prompt construction, we collect seed concepts (e.g. bicycle) and automatically construct several morpho-syntactically varying prompts (e.g. "a bicycle has . . . ") ( §2.1) for each concept. the prompts are used as inputs to i2d2. second, we employ constrained generation to control the style of text generated from the pre-trained lm at to mimic the style of generic statements( §2.2). third, a supervised critic is used to filter out false and ill-formed generations ( §2.3). finally, the language model is finetuned on its own high-quality generations selected by the critic in an iterative self-imitation learning setup ( §2.4). figure 2 illustrates the overall framework. ## experiments and results we describe results from our experiments comparing i2d2 with gpt-3, gpt-2 xl and genericskb in more detail below. figure 4 shows outputs sampled from these sources. ## conclusion we present i2d2-a novel framework for generating generic knowledge from language models using constrained decoding and self-imitation learning. i2d2, while using orders of magnitude fewer parameters, can still outperform gpt-3 at the task of generating high-quality generic statements. we also show that gen-a-tomic is higher-quality, larger-scale, and more diverse than the static gener-icskb dataset. i2d2 provides on-demand access to generic knowledge that can bridge the gap in commonsense knowledge, often observed in even the largest lms available today.