Crystal: Introspective Reasoners Reinforced with Self-Feedback (2023)

Abstract: Extensive work has shown that the performance and interpretability of commonsense reasoning can be improved via knowledge-augmented reasoning methods, where the knowledge that underpins the reasoning process is explicitly verbalized and utilized. However, existing implementations, including “chain-of-thought” and its variants, fall short in capturing the *introspective* nature of knowledge required in commonsense reasoning, and in accounting for the mutual adaptation between the generation and utilization of knowledge. We propose a novel method to develop an introspective commonsense reasoner, **Crystal**. To tackle commonsense problems, it first introspects for knowledge statements related to the given question, and subsequently makes an informed prediction that is grounded in the previously introspected knowledge. The knowledge introspection and knowledge-grounded reasoning modes of the model are tuned via reinforcement learning to mutually adapt, where the reward derives from the feedback given by the model itself. Experiments show that Crystal significantly outperforms both the standard supervised finetuning and chain-of-thought distilled methods, and enhances the transparency of the commonsense reasoning process. Our work ultimately validates the feasibility and potential of reinforcing a neural model with self-feedback.

## introduction commonsense reasoning poses unique challenges to neural reasoning models. the underlying knowledge that grounds the reasoning process is often obscure and inexplicable, even to humans as we mainly rely on intuitive inference for such problems @xcite . this is in stark contrast with multi-step logical reasoning (e.g., figure 1 : top: crystal performing introspective reasoning on a commonsense question. the model first uses its knowledge introspection mode to generate relevant knowledge statements, then invokes a knowledgegrounded reasoning mode to predict an answer based on the introspected knowledge. bottom: chain-ofthought prompting on the same question (generated by text-davinci-003 with original few-shot prompts in @xcite ). the intermediate steps fail to provide meaningful insight into the reasoning process. math problems, logical deductions), where the reasoning process consists of explicit and closedworld deduction steps. chain-of-thought (cot) @xcite and its variants have been successful in multi-step logical reasoning, yet their effectiveness on commonsense reasoning is marginal, largely due to the lack of the above observation when designing their few-shot prompts. nevertheless, generating the reasoning process is still instrumental for commonsense reasoning, as it improves both performance and interpretability of neural models @xcite . for such knowledge-augmented reasoning approach, two components are indispensable: (1) introspecting for relevant, high-quality knowledge, and (2) effectively and faithfully utilizing the knowledge to make informed final predictions. our key insight is that these two components are deeply adaptive to each other: knowledge introspection should aim to produce knowledge that would be most beneficial to grounding the subsequent reasoning, and knowledge-grounded reasoning should learn to best leverage the previously introspected knowledge. existing literature does not comprehensively optimize these two components and the bi-directional interaction between them, and comes with additional complications. knowledge-augmented reasoning methods largely employ task-specific engineering for knowledge generation and are thus difficult to generalize to unseen domains. as for cot and its variants, the reasoning chains hardly provide meaningful information due to deficiency in their prompt design (figure 1 ). we aim to systematically address the above considerations and build a strong, interpretable and generalizable model for commonsense reasoning. the introspective reasoner we develop, named crystal, tackles commonsense problems by the following (illustrated in figure 1 ): it first invokes a knowledge introspection mode to generate knowledge statements related to the given question, and subsequently invokes a knowledge-grounded reasoning mode that ingests both the question and the previously introspected knowledge to predict an answer. crystal is trained with reinforcement learning (rl) to improve the synergy between the reasoning paths and the final predictions. the knowledge introspection mode of the model is trained with ppo @xcite to optimize a reward function that characterizes if the generated knowledge can fix prediction errors made by the knowledge-grounded reasoning mode of the model. in this sense, crystal is reinforced with selfgenerated feedback. concurrently, the knowledgegrounded reasoning mode evolves to better utilize the introspected knowledge statements for more accurate predictions. these two learning objectives are harmonized through a novel interleaved optimization schedule, echoing the principles of the em algorithm @xcite . we employ a two-stage training process: the rl training stage is preceded by a supervised training stage, where crystal acquires preliminary skills to generate and utilize knowledge by imitating a larger lm (e.g., gpt-3). experimental results on 25 commonsense qa benchmarks (10 seen, 15 unseen) show that crys-tal not only enhances performance within fixed model sizes, but also amplifies the interpretability of the reasoning process. crystal outper-forms direct qa models finetuned with standard supervised learning and the same data, improving absolute accuracy by 1.5% to 2.5% on different model sizes, and showcases good generalization to unseen benchmarks. this highlights the benefits of introspective reasoning over direct inference. additionally, crystal substantially outperforms models distilled from cot produced by large lms. through crystal, we illustrate the potential and viability of reinforcing neural reasoning models with self-feedback. an additional benefit of our approach is the memory and time-efficient implementation of ppo via model sharing, which allows this state-of-the-art rl algorithm to be applied to larger models with given amount of resources. ## method we will first introduce the concept of introspective reasoning ( §2.1), followed by a description of our introspective reasoner, crystal, including its basic functionality ( §2.2), training objectives ( §2.3), adaptation of the rl algorithm and efficiency improvements ( §2.4), and the design of model training process ( §2.5, §2.6). ## experimental setup datasets. to promote generalization, we train crystal on 10 datasets (following @xcite ): openbookqa @xcite , arc (easy and hard splits) @xcite , ai2science (elementary and middle splits) @xcite , commonsenseqa @xcite , qasc @xcite , physicaliqa @xcite , socialiqa @xcite , and winogrande @xcite . we use the development set of these datasets to evaluate model performance (i.e., seen evaluation). for unseen evaluation, we use the development set of 15 additional datasets: com2sense @xcite , sciq @xcite , quarel @xcite , quartz @xcite , cycic, comve @xcite , winograd schema challenge @xcite , copa @xcite , numersense (lin et al., 2020), prost (aroca-ouellette et al., 2021), swag @xcite , hellaswag @xcite , codah @xcite , story cloze test @xcite , and αnli @xcite . see models. similar to @xcite , we initialize crystal with t5 @xcite . the value regression head is initialized from scratch at the beginning of stage ii training. we experiment with three model sizes: t5-large, t5-3b, and t5-11b. we train models on v100 gpus (8 for t5-large, 16 for t5-3b, and 64 for t5-11b), with the huggingface transformers and accelerate libraries @xcite . see baselines. we primarily compare crystal with models trained on the same datasets using the standard qa objective (i.e., equation 1 ), referred as "direct qa". these models are also based on the pretrained t5 of the three sizes above. additionally, we compare our model to rainier @xcite and several cot-distilled models. among these, fine-tune-cot @xcite use zero-shot reasoning chains elicited from text-davinci-002 to finetune smaller variants of gpt-3; scotd @xcite employs a similar distillation strategy, whereas the teacher model is code-davinci-002 and the target model is opt up to 1.3b parameters; scott @xcite proposes additional techniques to improve the reasoning integrity, including a contrastive decoding method to elicit more consistent reasoning chains from the teacher model and a counterfactual reasoning method to train the target model, and yet does not enable full bidirectional adaptation between the reasoning process and the final prediction, as our method does. it is worth noting that these cot-distilled models are often trained on specific datasets, so we only present their reported performance on the datasets they were trained on. we also report the existing sota performance achieved by non-retrieval methods on each seen dataset. 2 we exclude retrieval-based methods for fair comparison, because crystal does not rely on retrieval from extra sources. ## conclusion we develop a method to build introspective reasoners that achieves superior performance and good interpretability on commonsense reasoning tasks. compared with prior literature, our method comprehensively accounts for the introspective nature of knowledge required in commonsense reasoning, and the mutual adaptation of knowledge introspection and knowledge-grounded reasoning. our approach highlights the feasibility and benefit of training neural models with self-feedback.