Mathematical Word Problem Generation from Commonsense Knowledge Graph and Equations (2021)

Abstract: There is an increasing interest in the use of mathematical word problem (MWP) generation in educational assessment. Different from standard natural question generation, MWP generation needs to maintain the underlying mathematical operations between quantities and variables, while at the same time ensuring the relevance between the output and the given topic. To address above problem, we develop an end-to-end neural model to generate diverse MWPs in real-world scenarios from commonsense knowledge graph and equations. The proposed model (1) learns both representations from edge-enhanced Levi graphs of symbolic equations and commonsense knowledge; (2) automatically fuses equation and commonsense knowledge information via a self-planning module when generating the MWPs. Experiments on an educational gold-standard set and a large-scale generated MWP set show that our approach is superior on the MWP generation task, and it outperforms the SOTA models in terms of both automatic evaluation metrics, i.e., BLEU-4, ROUGE-L, Self-BLEU, and human evaluation metrics, i.e., equation relevance, topic relevance, and language coherence. To encourage reproducible results, we make our code and MWP dataset public available at https://github.com/tal-ai/MaKE_EMNLP2021 .

## introduction a mathematical word problem (mwp) is a coherent narrative that provides clues to the underlying correct mathematical equations and operations between variables and numerical quantities @xcite . mwps challenge a student from a wide range of skills such as literacy skills for understanding the question, analytical skills for recognizing the problem type and applying arithmetical operators @xcite . ## learning from commonsense knowledge and equations our objective is to automatically generate a significant number of diverse mwps in students' real-life scenarios from valid equations. in addition, we support the personalized generation in which students (or teachers) can determine the story plots of mwps by specifying topics and mapping relations between variables and entities (i.e., "x: chicken, y: rabbits", "x: apple, y: banana", etc.). a topic indicates a type of real-world scenarios, such as animals, fruits, etc. as shown in figure 1 , we adopt the encoderdecoder generation framework. the input includes a set of equations and a knowledge graph with a specific topic. we construct levi graphs @xcite from symbolic equations and the cskg respectively (see section 3.1). after that, we employ ggnns to extract the full graph structure information about equations and real-life story plots (see section 3.2). then, we generate target sentence by a conditional vae with a self-planning module (see section 3.3). the self-planning module enables the decoder to pay different portions of attention to the equations and the cskg. please note that in this paper, we focus on generating mwps with linear equations of two variables without any constraint. our framework can be easily generalized into mwps with different numbers of variables with little modification. ## experiments in this work, we crawled 5,447 mwps of linear equations from a third-party website, and each mwp consists of two unknown variables and two equations. it covers 119 topics and the average length of an mwp is 62 words. in one cskg, the average number of entities is 17.067 and the average number of edges is 29.102. we randomly select 544 of them as our validation set, and 546 of them as our gold-standard test (gt) set. please note that different from previous work of automatically solving the mwps such as mawps @xcite and mathqa @xcite , we focus on the generation task of mwps of more than one linear equations in students' real-life scenarios by using topics in our cskg. both mawps and mathqa datasets do not contain mwps that have two or three equations and variables. furthermore, there is no explicit topics associated with the mwps in these publicly available mwp datasets. we use following evaluation metrics: (1) bleu-4: the 4-gram overlap score against gold-standard sentences @xcite ; (2) meteor: n-gram overlap with paraphrase and languagespecific considerations @xcite ; (3) rouge-l: the overlap of longest common subsequence between candidate and goldstandard sentences @xcite ; (4) self-bleu: the diversity measurement of averaging bleu scores of four generated mwp pairs given the same input @xcite . meanwhile, we conduct two human evaluation studies to comprehensively evaluate the quality of the generated mwps. first, we ask three evaluators to rate from the following aspects ranging from 1 to 3: (1) equation relevance: how relevant is mwp with respect to the input equations? (2) topic relevance: how relevant is mwp with respect to the given topic? and (3) language coherence: whether the mwp is coherent and well-organized. we use the average scores from three human evaluators as our final results. before training, in order to ensure that each question is answerable, we first use sympy's equation solver 3 to solve all the algebraic equations in the we compare our make against several strong baselines: (1) the template based method, i.e., template; (2) conditional vae that captures the diversity in the encoder and uses latent variables to learn a distribution over potential intents, i.e., cvae @xcite ; (3) an rnn-based seq2seq model with equation-topic fusion mechanism and entity-enforced loss, i.e., magnet (zhou and huang, 2019); (4) the sota pre-trained language model with a shared transformer network and self-attention masks, i.e., unilm @xcite ; and (5) a standard transformer-based seq2seq model, i.e., transformer @xcite . more details are provided in appendix a.4. please note that we do not select rewriting based approaches as the baselines in this work. this is because rewriting based approaches require a very large pre-stored question bank and it only works when the input equations are matched in the question bank. ## conclusion in this paper, we presented a neural encodingdecoding architecture for mwp generation. comparing with the existing nqg algorithms, the advantages of our make are: (1) it extracts intrinsic representations of both the equation based symbolic graph and the cskg; (2) it automatically selects and incorporates information from equations and knowledge graphs during the decoding process; and (3) it is able to generate relevant, coherent and diverse mwps in students' real-life scenarios. experimental results on real-world educational mwp data sets demonstrate that make outperforms other sota nqg approaches in terms of both automatic evaluation metrics and human evaluation metrics. in the future, we plan to explore the mwp generation problems for more mathematical variables with high-order operations, and explore the method to incorporate commonsense knowledge from publicly available cskg like conceptnet or wikipedia.