An Enhanced Knowledge Injection Model for Commonsense Generation (2020)

Abstract: Commonsense generation aims at generating plausible everyday scenario description based on a set of provided concepts. Digging the relationship of concepts from scratch is non-trivial, therefore, we retrieve prototypes from external knowledge to assist the understanding of the scenario for better description generation. We integrate two additional modules into the pretrained encoder-decoder model for prototype modeling to enhance the knowledge injection procedure. We conduct experiment on CommonGen benchmark, experimental results show that our method significantly improves the performance on all the metrics.

## introduction recently, commonsense reasoning tasks such as swag @xcite , commonsenseqa @xcite and commongen @xcite are presented to investigate the model's ability to make acceptable and logical assumptions about ordinary scenes in our daily life. swag requires to infer the probable subsequent event based on the given textual description of an event. commonsenseqa focuses on commonsense question answering that collects commonsense questions at scale by describing the relation between concepts from conceptnet. different from these discriminative tasks, commongen is a generation task that not only needs to use background commonsense knowledge to conduct relational reasoning, but also compositional based generation capability. considering commongen requires model to reason about a potentially infinite number of novel combinations of concepts, there is a big gap between human performance and automatic generation models even current most powerful pretrained models @xcite @xcite . this demonstrates that dig the combination relationship between these concepts from scratch does not suffice for generative commonsense reasoning. @xcite shows that the task of generating complex outputs through editing existing outputs can be easier than generating complex outputs from scratch. this inspires us to use the retrieve-and-edit framework to retrieve a prototype with these concepts to enhance the commonsense generation task. there are two main advantages of the retrieve-and-edit framework. first, the knowledge of these concepts are implicit and compositional which makes it hard to find out a plausible scene from the concepts combination, such as "feel" and "pierce" appear in the generation of bart in in this work, we proposed to retrieve a prototype from a knowledge corpus, and use this prototype to guide the commonsense generation procedure. we utilize english wikipedia and daily scenario datasets ## model in this section, we introduce our retrieve-and-edit framework based eki-bart g θ with parameter θ that extracts prototype @xmath0, o 2 , • • • , o no ) from external text knowledge corpus and edits the prototype following the requirement of @xmath1, • • • , c nc ) to improve the commonsense generation of target @xmath2, • • • , t nt ). the overall framework of our proposed model is shown in figure 1 . ## experiment in this section, we conduct experiments to prove the effectiveness of our proposed approach. to dig into our approach, we perform ablation studies to explore the different effects of scaling module and prototype position indicator. ## conclusion and future work in this paper, we have proposed a pretraining enhanced retrieve-and-edit model for commonsense generation. the key of commongen is to identify the priority of the scene based on the concept combination, we have scaling module to softly reduce the impact of prototype noises on generation and prototype position indicator to help decoder better learn the prototype scenario. our retrieve-and-edit model with in-domain and out-of-domain dataset both achieve better performance. in future, we plan to build the relationship of these concepts in a more structure manner.