Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints (2021)

Abstract: We study the problem of generating arithmetic math word problems (MWPs) given a math equation that specifies the mathematical computation and a context that specifies the problem scenario. Existing approaches are prone to generating MWPs that are either mathematically invalid or have unsatisfactory language quality. They also either ignore the context or require manual specification of a problem template, which compromises the diversity of the generated MWPs. In this paper, we develop a novel MWP generation approach that leverages i) pre-trained language models and a context keyword selection model to improve the language quality of generated MWPs and ii) an equation consistency constraint for math equations to improve the mathematical validity of the generated MWPs. Extensive quantitative and qualitative experiments on three real-world MWP datasets demonstrate the superior performance of our approach compared to various baselines.

## introduction math word problems (mwps) are an important type of educational resource that help assess and improve students' proficiency in various mathematical concepts and skills @xcite ). an mwp usually has a corresponding underlying math equation that students will need to identify by parsing the problem and then solve the problem using this equation. an mwp is usually also associated with a "context", i.e., the (often real-world) scenario that the math equation is grounded in, expressed in the question's text. the equation associated with an mwp is often exact and explicit, while the context of the mwp is more subtle and implicit. it is not immediately clear how the context information can be extracted or represented. mwp: joan found 70 seashells on the beach . she gave sam some of her seashells . she has 27 seashells . how many seashells did she give to sam ? equation: @xmath0 in this work, we study the problem of automatically generating mwps from equations and context, which is important for three reasons. first, an automatic mwp generation method can aid instructors and content designers in authoring mwp questions, accelerating the (often costly and laborintensive) mwp production process. second, an automated mwp generation method can generate mwps tailored to each student's background and interests, providing students with a personalized learning experience @xcite that often leads to better engagement and improved learning outcomes @xcite @xcite @xcite . third, an automated mwp generation method can potentially help instructors promote academic honesty among students. while new technologies create new learning opportunities, instructors have growing concerns of technologies that enable students to easily search for answers online without actually solving problems on their own @xcite . automatically generated mwps that are unique and previously unseen yet preserve the underlying math components can potentially reduce plagiarism. in addition to its educational utility, mwp generation is also technically challenging and interesting. an important consideration for mwp generation is controllability: in practice, human instructors or content designers often have clear preferences in the type of mwps they want to use. therefore, an mwp generation method should be able to generate mwps that are of high language quality and are textually and mathematically consistent with the given equations and contexts. to date, there exist limited literature on mwp generation. most prior works focus on automatically answering mwps, e.g., @xcite @xcite @xcite @xcite instead of generating them @xcite @xcite . existing mwp generation methods also often generate mwps that either are of unsatisfactory language quality or fail to preserve information on math equations and contexts that need to be embedded in them. see section 4 for a detailed discussion. ## methodology we formulate the task of controllable mwp generation as a conditional generation problem. in this paper, we work with datasets @xmath1, e i )} n @xmath2, equation) pairs where m i and e i represent mwp and its associated equation, respectively. in the remainder of the paper, we will remove the data point index to simplify notation. this setup assumes each mwp in our dataset is labeled with an underlying equation but its context is unknown. then, the mwp generation process can be described as in this work, we use a pre-trained language model (lm) as the generative model p θ , similar to the setup in @xcite . we choose lms over other approaches such as sequence-tosequence (seq2seq) models because they can be pre-trained on web-scale text corpora. pre-trained lms thus often generate high-quality text and generalizes well to out-of-domain words not present in the training data. under an lm, we can further decompose eq. 1 into to train p θ via fine-tuning the lm, we use the usual negative log-likelihood objective: the above training objective serves as a proxy that optimizes for language quality. however, it alone is unsatisfactory in 2 ways. first, there is no guarantee that the generated mwp is mathematically valid; even if it is, its solution may correspond to an equation that is different from the input equation @xcite . second, while the context c can be manually specified, i.e., as a set of keywords, it is unobserved during training and needs to be inferred from data through the costlyto-compute posterior distribution. in the remainder of this section, we introduce our novel approach to tackle these challenges. we first describe our equation consistency constraint that improves the generated mwp's mathematical consistency and then detail our context selection method that learns to extract the context in the form of a set of keywords from an mwp. figure 1 provides a high-level overview of our overall approach. ## experiments we now perform a series of experiments to validate the effectiveness of our proposed mwp generation approach. quantitatively, we compare our approach to several baselines on various automated language quality and mathematical consistency metrics. qualitatively, we showcase the capability of our approach in generating controllable, highquality mwps. datasets. we focus on mwp datasets in which each mwp is associated with a single equation and each equation contains a single unknown variable. therefore, we consider three such mwp datasets including arithmetic @xcite , mawps (koncel-kedziorski et al., 2016), and math23k @xcite . other popular mwp datasets such as algebra @xcite , dolphin18k @xcite and mathqa 1 @xcite contain mwps with multiple equations and many variables, which are challenging to generate even for humans. we leave the more challenging case of generating multivariable, multi-equation mwps to future work. ## conclusions and future work in this paper, we developed a controllable mwp generation approach that (i) leverages pre-trained language models to improve language quality of the generated mwp, (ii) imposes an equation consistency constraint to improve mathematical consistency of the generated mwp, and (iii) includes a context selector that sets the context (in the form of a set of keywords) to use in the generation process. experimental results on several real-world mwp datasets show that, while there is plenty of room for improvement, our approach outperforms existing approaches at generating mathematically consistent mwps with high language quality. automatically generating mwps remains a challenging problem and our work opens up many avenues for future work. first, our study is limited to the case of simple mwps, each with a single equation and variable. while results are encouraging, our approach does not generalize well to the more challenging case when the input consists of multiple, complex equations. in these cases, we need more informative representations of the input equations @xcite . second, there is no clear metric that can be used to evaluate the generated mwps, especially their mathematical validity. it is not uncommon when a generated mwp with high scores under our metrics is either unanswerable or inconsistent with the input equation. therefore, future work should also focus on developing metrics for better evaluation of generated mwps' mathematical validity. last but not least, while we focus on 2 control elements (equation, context), an interesting future direction is to add more control elements to the generation process such as question difficulty and linguistic complexity.