Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022)

Abstract: Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent.We present Rainier, or Reinforced Knowledge Introspector, that learns to generate contextually relevant knowledge in response to given questions. Our approach starts by imitating knowledge generated by GPT-3, then learns to generate its own knowledge via reinforcement learning where rewards are shaped based on the increased performance on the resulting question answering. Rainier demonstrates substantial and consistent performance gains when tested over 9 different commonsense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.

## introduction commonsense is a significant challenge for modern nlp models, due to the obscurity of underlying knowledge that grounds the reasoning process. while humans are generally able to introspect the underlying reasons for their conclusion @xcite , neural models lack the capability to verbalize the premises leading to their prediction. this hinders models' performance and robustness on commonsense tasks, and makes it difficult to inspect their point of failure. recent research has demonstrated that relevant knowledge can provide useful context for approaching commonsense tasks. yet these methods either retrieve from indomain knowledge bases @xcite ) that do not have good coverage over commonsense, or generate knowledge from neural models @xcite @xcite , which often need domain-specific engineering and very large models (e.g. @xcite ). it is still an open challenge to systematically find high-quality knowledge. in this work, we use a novel, reinforcementlearning-based method to develop rainier, a generative neural model that can introspect the underlying knowledge for reasoning about commonsense questions. as illustrated in fig 1 , rainier is trained to generate knowledge that are both fluent natural language statements, and useful prompts that optimize the performance of a generic question answering (qa) model. our model (1) demonstrates strong generalization to unseen benchmarks without additional engineering effort (e.g. finetuning), (2) produces commonsense knowledge of high quality and diversity, and (3) is substantially smaller in size compared to gpt-3, the best knowledge source reported so far. to train rainier, we optimize knowledge introspection for the resulting qa, instead of direct supervision, because there are usually no gold knowledge labels on commonsense datasets. in order to ensure that our model learns to generate generically useful knowledge for a broad range of qa models, we train only rainier, the knowledge introspector, without finetuning the qa model. since our desired knowledge are sequences of discrete, non-differentiable word tokens, we adapt a reinforcement learning algorithm, proximal policy optimization (ppo) @xcite , to optimize the knowledge introspector. specifically, the reward is defined as the effect of rainier-generated knowledge on the qa model's prediction. we train rainier in a multitask setting on 8 commonsense qa datasets -encompassing general, scientific, physical, and social commonsense -to equip the model with better generalization to unseen benchmarks. experiments show that rainier substantially improves the performance of qa models on 9 commonsense benchmarks (5 datasets seen during training and 4 unseen datasets), and gives larger and more consistent gains than a few-shot gpt-3 @xcite despite being 16x smaller in parameter size. it also boosts the performance on top of those qa models that it is not trained against, indicating that it generates generically useful knowledge instead of merely hacking into the reward given by a single qa model. knowledge generated by rainier can even boost a qa model that is 4x larger than it, showing the promise of using modelgenerated knowledge as a complement to model scaling in making progress in commonsense reasoning. our analyses show that the knowledge generated by rainier are of high quality, and are diverse in terms of domain (e.g. scientific, social), relation expressed (e.g. part of, member of, purpose), and syntactic property (e.g. negation, comparison). the effect of these knowledge on the qa model also aligns well with human judgments. the success of rainier shows that moderately-sized models can serve as source of high-quality and useful commonsense knowledge that facilitates reasoning. we publicly release the code, the trained rainier model, and the commonsense datasets extended with knowledge generated by rainier. algorithm 1 training rainier input initial policy model θ0, initial value model ϕ0, pretrained qa model ψqa dimit ← get silver knowledge on dseen from gpt-3. θimit ← optimize θ0 with eqn 2 from dimit. ▷ section 2.1 θrainier ← reinforcedlearning(dseen, θimit, ϕ0, ψqa) ▷ section 2.2 procedure reinforcedlearning(dseen, θ, ϕ, ψqa) θold ← θ, ϕold ← ϕ for @xmath0, 2, . . . do sample a minibatch from dseen. for @xmath1, 2, . . . , s do compute lppo on the minibatch with eqn 3. optimize θ and ϕ with lppo for one step. common approaches only train supervised qa models. as a complement, we train a separate model, which we refer to as rainier, that can introspect question-specific knowledges that are useful to prompt a fixed qa model. rainier is a sequence-to-sequence language model, p k (k|q; θ), and we expect it to generate knowledge statements (k's) in response to the given question (q). however, the challenge is that we have no gold knowledge labels as supervision. training. since we do not have gold knowledge to train rainier, we obtain this model by finetuning a pretrained language model in two stages: (i) imitation learning, and then (ii) reinforcement learning. in stage i ( §2.1), we get silver knowledge labels on some datasets from gpt-3, and teach our model to imitate this knowledge-generating gpt-3. this equips our model with the basic functionality of knowledge generation. in stage ii ( §2.2), we use reinforcement learning to continue training the model obtained in stage i to make the generated knowledge more useful while staying fluent and meaningful. specially, we set the reward to be the effect of the generated knowledge on the prediction made by a fixed, generic qa model. we obtain silver knowledge and train rainier on the union of multiple qa datasets (which are considered seen during training), i.e. d @xmath2, where the generic qa model we use may or may not have been trained on these seen datasets. the complete training process is outlined in algorithm 1. inference. the effectiveness of rainier is evaluated against a set of unseen qa datasets, d unseen , in addition to the seen datasets. note that rainier is not trained on any unseen datasets, which means we neither get silver knowledge, nor do imitation learning or reinforcement learning on them. the generic qa model we use was not trained on any unseen datasets as well. we discuss details of inference in §2.3. ## conclusion we introduced rainier, a neural model that can introspect for relevant knowledge on a broad range of commonsense question answering tasks. rainier is trained with a novel adaption of reinforcement learning, and does not need gold knowledge labels that are difficult to obtain. knowledge generated by rainier can serve as useful prompts that improves the performance of qa models on both seen and unseen benchmarks, and outperform knowledge elicited from a few-shot gpt-3 which is 16x bigger. rainier generates knowledge in the form of natural language statements that are fluent, meaningful, high-quality, and diverse in terms of domain and relation; furthermore, the effect of these knowledge on the qa model is found to align well with human judgments.