Generate & Rank: A Multi-task Framework for Math Word Problems (2021)

Abstract: Math word problem (MWP) is a challenging and critical task in natural language processing. Many recent studies formalize MWP as a generation task and have adopted sequence-to-sequence models to transform problem descriptions to mathematical expressions. However, mathematical expressions are prone to minor mistakes while the generation objective does not explicitly handle such mistakes. To address this limitation, we devise a new ranking task for MWP and propose Generate & Rank, a multi-task framework based on a generative pre-trained language model. By joint training with generation and ranking, the model learns from its own mistakes and is able to distinguish between correct and incorrect expressions. Meanwhile, we perform tree-based disturbance specially designed for MWP and an online update to boost the ranker. We demonstrate the effectiveness of our proposed method on the benchmark and the results show that our method consistently outperforms baselines in all datasets. Particularly, in the classical Math23k, our method is 7% (78.4% to 85.4%) higher than the state-of-the-art. Code could be found at https://github.com/huawei-noah/noah-research .

## introduction solving math word problems (mwp) @xcite is an important and fundamental task in natural language processing (nlp), which requires to provide a solution expression given a mathematical problem description, as illustrated in number-mapped mwp problem a project is completed in num0 days by num1 workers. if it takes num2 days to complete, how many workers will it take? solution num0 * num1 / num2 from source texts to target expressions. these studies have proposed numerous advanced techniques to improve the mwp solver, but their performance is still unsatisfactory yet. we argue that it is not sufficient to model mwp as only a generation task, because there is a significant difference between mathematical expressions and natural language sequences: one minor mistake in a mathematical expression will change the whole semantic thus lead to a wrong answer, whereas natural language is more robust to such minor mistakes. the objective function of the generation task is to maximize generation likelihood on ground-truth expressions, which does not have an explicit strategy to make the model learn to distinguish between ground-truth and expressions that have minor mistakes. in addition, previous works @xcite @xcite find that the performance of generation models degrades fast as the expression gets longer. to handle the above problems, we propose generate & rank, a multi-task framework for mwp, which introduces a new ranker to explicitly distinguish between correct and incorrect expressions. specifically, our framework includes two modules: a generator and a ranker. the former is designed to generate candidate expressions given a problem text and the latter aims to rank the candidate expressions. they are built based on an encoderdecoder model and are jointly trained with generation loss and ranking loss. in this work, we build our model based on bart @xcite , a widely used pre-trained language model that achieves sota performance on various sequenceto-sequence tasks @xcite . during multi-task training, expressions produced by the generator are used to construct an expression bank and train the ranker, in which way the model can learn from its own mistakes. to construct more informative candidates for the ranker, we specially design tree-based disturbance for mwp. we also introduce an online update mechanism to generate a new set of candidate expressions at each training epoch. the overall training procedure is in an iterative manner, in which the ranker and generator continue to enhance each other. to evaluate the effectiveness of the proposed model, we conduct extensive experiments on the datasets of math23k @xcite and mawps @xcite . the results show that our model outperforms typical baselines. particularly, we obtain an improvement of 7% in the math23k dataset that is extensively studied. moreover, we do ablation study and model analysis, which shows that (1) joint training improves the performance of the generator and ranker over separate training; (2) both strategies of constructing candidate expressions and online updating are important to the success of the ranker. we also find that with the ranker, our model achieves a large improvement in generation of long expressions. the contributions of our work are two-fold: (1) we propose generate & rank, a new multi-task framework to train a pre-trained language model for math word problem solving. to construct informative candidate expressions for the ranker, we propose two effective generation methods and also introduce an online update strategy. (2) experiments show that our proposed model consistently outperforms the state-of-the-art models and achieves a significant improvement on the math23k dataset. ## methodology we propose generate & rank, a bart-based multitask framework for math word problems. our model consists of a generator and a ranker, which share a bart model and are jointly trained with a generating task and ranking task. the objective of generating is to generate expressions given a math word problem. we also add a ranking task so that the model can select a correct expression from a set of candidates. we construct an expression bank to provide training examples for the ranker. figure 1 shows our proposed framework and we introduce details for each task and the whole framework in the following sections. ## conclusion and future work we propose generate & rank, a new multi-task framework for math word problems. specifically, our model has a generator and a ranker which enhance each other with joint training. we also use tree-based disturbance and online update to further improve the performance. the experimental results on the benchmark show that our work consistently outperforms baselines in all datasets. in future work, we will explore the generation and ranking framework to other tasks like summarization and translation.