WARM : A Weakly (+ S emi) Supervised Math Word Problem Solver (2022)

Abstract: Solving math word problems (MWPs) is an important and challenging problem in natural language processing. Existing approaches to solving MWPs require full supervision in the form of intermediate equations. However, labeling every MWP with its corresponding equations is a time-consuming and expensive task. In order to address this challenge of equation annotation, we propose a weakly supervised model for solving MWPs by requiring only the final answer as supervision. We approach this problem by first learning to generate the equation using the problem description and the final answer, which we subsequently use to train a supervised MWP solver. We propose and compare various weakly supervised techniques to learn to generate equations directly from the problem description and answer. Through extensive experiments, we demonstrate that without using equations for supervision, our approach achieves accuracy gains of 4.5% and 32% over the current state-of-the-art weakly-supervised approach, on the standard Math23K and AllArith datasets respectively. Additionally, we curate and release new datasets of roughly 10k MWPs each in English and in Hindi (a low-resource language). These datasets are suitable for training weakly supervised models. We also present an extension of our model to semi-supervised learning and present further improvements on results, along with insights.

## introduction a math word problem (mwp) is a numerical problem expressed in natural language (problem de-scription), that can be transformed into an equation (solution expression), which can be solved to obtain the final answer. in we propose a novel two-step weakly supervised technique to solve mwps by making use only of the weak supervision, in the form of answers. in the first step, using only the answer as supervision, we learn to generate equations for questions in the training set. in the second step, we use the generated equations along with answers to train any state-of-the-art supervised model. we illustrate the effectiveness of our weakly supervised approach on our newly curated reasonably more recently, neural network-based mwp solving methods have been proposed @xcite @xcite @xcite @xcite @xcite . these employ an encoder-decoder architecture and train in an end-to-end manner without the need for hand-crafted rules or templates. @xcite were the first to propose a sequenceto-sequence (seq2seq) model, viz., deep neural solver, for solving mwps. they employ an rnn-based encoder-decoder architecture to directly translate the problem text into equation templates and also release a high-quality large-scale dataset, math23k, consisting of 23,161 mwps in chinese. @xcite and @xcite propose tree-structured decoding that generates the syntax tree of the equation in a top-down manner. in addition to applying tree-structured decoding, @xcite propose a graph-based encoder to capture relationships and order information among the quantities. for a more comprehensive review on automatic mwp solvers, readers can refer to a recent survey paper @xcite . unlike all the previous works that require equations for supervision, @xcite propose a weakly supervised method for solving mwps, where the answer alone is required for training. their approach attempts to generate the equation tree in a rule based manner so that the correct answer is reached. they then train their model using the fixed trees. with the same motivation. we also propose a novel weakly supervised model, warm, (c.f., section 4) for solving mwps using only the final answer for supervision. we show how warm can be extended to semi-supervised joint learning in the presence of weak answer-level supervision in conjunction with some equation-level supervision. further, we empirically demonstrate that warm outperforms @xcite on all the datasets. we also took insights from @xcite , @xcite , @xcite , @xcite , @xcite , @xcite and @xcite for handling mathematical data in two different languages. this paper is organized as follows. in section 3, we set the premise for our approach by describing the new datasets (ew10k and hw10k) for weak supervision that we release. in section 4, we describe our weakly supervised approach warm and its semi-supervised extension warm-s. in section 5, we present the experimental setup whereas in section 6 we delve into the results and its analysis before concluding in section 7. ## dataset currently, there does not exist any sufficiently large english dataset for single and simple equation mwps. while there exists an english dataset @xcite with sufficiently large mwps, the questions in the dataset are meant to be evaluated in a multiple choice question (mcq) manner. also, the equation associated with each word problem in this dataset is significantly more complex and requires several binary and unary operators. on the other hand, math23k @xcite is in chinese and dolphin18k @xcite ) contains mostly multi-variable word problems. to address these gaps, we curate a new english mwp dataset, viz., ew10k foot_0 consisting of 10227 word problem instances (each associated with a single equation) that can be used for training mwp solver models in a weakly supervised manner. we crawled ixl foot_1 to obtain mwps for grades vi until x. these word problems involve a wide variety of mathematical computations ranging from simple addition-subtraction to much harder mensuration and probability problems. the dataset consists of 10 different types of problems, spanning 3 tiers of difficulty. we also annotate the dataset with the target unit. the exact distributions are presented in figure 1 . we similarly created a mwp dataset in hindi 2a low resource language. it consists of 9,896 question answer pairs. to the best of our knowledge, this is the first mwp dataset of such size in hindi. 4 our approach: warm we propose a weakly supervised model, warm, for solving the mwp using only the answer for supervision. it is a two-step cascaded approach for weakly supervised mwp solving. for the first step, we propose a model that predicts the equation, given a problem text and answer. this model uses reinforcement learning to search the space of possible equations, given the question and the correct answer only. the answer acts as the goal of the agent and the search is terminated either when the answer is reached or when the equation length exceeds a pre-defined length (this is required, else the search space would be infinitely large). the model is designed to be a two layer bidirectional gru @xcite encoder and a decoder network with fully connected units (described in section 4.3). we refer to this model as warm. note that this model requires an answer to determine when to stop exploring. since we ultimately want a model which should only take the problem statement as input and generate the answer (by generating the correct equation), this model alone is insufficient for evaluation. using this model, we create a noisy equation-annotated dataset from the weakly annotated training dataset (the training dataset has answers since it is weakly supervised). we use only those instances to create the dataset for which the equation generated by the model yields the correct answer. note that the equations are noisy, since there is no guarantee that the generated equation will be the shortest or even correct. in the second step, we use this noisy data for supervised training of a state-of-the-art model. the trained supervised model is finally used for evaluation. for simplicity, we provide a summary of notations in section 1 in supplementary. figure 3: architecture for generating equation tree in warm. ## experimental setup in this section, we report details of the experiments on four datasets to examine the performance of the proposed weakly supervised model warm and its semi-supervised extension warm-s. we present comparisons with various baselines as well as with fully supervised models. ## conclusion we have proposed a two step approach to solving math word problems, using only the final answer for supervision. our weakly supervised approach, warm, achieves a reasonable accuracy of 56.0 on the standard math23k dataset even without leveraging equations for supervision. we also curate and release large scale mwp datasets, ew10k, in english and hw10k, in hindi. we observed that the results are encouraging for simpler mwps. we also present the benefits of incorporating a semisupervised extension to warm.