Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022)

Abstract: Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.

## introduction an important desideratum of natural language generation (nlg) is to produce outputs that are not only correct but also diverse @xcite . the term "diversity" in nlg is defined as the ability of a generative model to create a set of possible outputs that are each valid given the input and vary as widely as possible in terms of content, language style, and word variability @xcite . this research problem is also referred as one-to-many generation @xcite @xcite . diversity in nlg has been extensively studied for various tasks in the past few years, such as machine translation @xcite and paraphrase § codes of our model and baselines are available at https://github.com/dm2-nd/mokge . ## proposed method problem formulation. in this paper, we focus on diversifying the outputs of generative commonsense reasoning (gcr) tasks, e.g. commonsense explanation generation and abductive commonsense reasoning. these tasks require one-to-many generation, i.e., creating a set of reasonable outputs that vary as widely as possible in terms of con-tents, language style and word variability. formally, given a source input x, our goal is to model a conditional distribution for the target outputs p(y|x) that assigns high values to {p( meanwhile, the outputs {y 1 , • • • , y k } are expected to be diverse with each other in terms of contents. existing diversity-promoting methods only varied the language styles and failed to perform different knowledge reasoning to generate diverse contents @xcite @xcite . here, incorporating commonsense kg is essential for the generative reasoning (gr) tasks because the kg cannot only augment the limited information in the input text, but also provide a rich searching space for knowledge reasoning. therefore, we propose to employ commonsense kg to play the central role of performing diverse knowledge reasoning, then use different sets of selected concepts to produce diverse outputs. model outline. our model has two major components: (i) a knowledge graph (kg) enhanced generative reasoning module to reasonably associate relevant concepts and background into the generation process, and (ii) a mixture of expert (moe) module to diversify the generation process and produce multiple reasonable outputs. ## future directions improving content diversity in nlg. most of the existing diversity-promoting work has focused on improving syntactic and lexical diversity, such as different language style in machine translation @xcite and word variability in paraphrase generation @xcite . nevertheless, methods for improving content diversity in nlg systems have been rarely studied in the existing literature. we believe that generating diverse content is one of the most promising aspects of machine intelligence, which can be applied to a wide range of real-world applications, not only limited to commonsense reasoning. besides, leveraging knowledge graph is not the only way to promote content diversity as it is a highly knowledge-intensive task. many existing knowledge-enhanced methods @xcite can be used to acquire different external knowledge for producing diverse outputs, e.g., taking different retrieved documents as conditions for generator. designing neural diversity metrics. in spite of growing interest in nlg models that produce diverse outputs, there is currently no principled neu-ral method for evaluating the diversity of an nlg system. as described in @xcite , existing automatic diversity metrics (e.g. self-bleu) perform worse than humans on the task of estimating content diversity, indicating a low correlation between metrics and human judgments. therefore, neural-based diversity metrics are highly demanded. intuitively, the metrics should include computational comparisons of multiple references and hypotheses by projecting them into the same semantic space, unlike metrics for evaluating the generation quality, e.g., bertscore @xcite and bleurt @xcite , which only measures the correlation between a pair of reference and hypothesis. ## conclusions in this paper, we proposed a novel method that diversified the generative reasoning by a mixture of expert strategy on commonsense knowledge graph. to the best of our knowledge, this is the first work to boost diversity in nlg by diversifying knowledge reasoning on commonsense knowledge graph. experiments on two generative commonsense reasoning benchmarks demonstrated that mokge outperformed state-of-the-art methods on diversity, while achieving on par performance on quality.