MWP - BERT : Numeracy-Augmented Pre-training for Math Word Problem Solving (2022)

Abstract: Math word problem (MWP) solving faces a dilemma in number representation learning. In order to avoid the number representation issue and reduce the search space of feasible solutions, existing works striving for MWP solving usually replace real numbers with symbolic placeholders to focus on logic reasoning. However, different from common symbolic reasoning tasks like program synthesis and knowledge graph reasoning, MWP solving has extra requirements in numerical reasoning. In other words, instead of the number value itself, it is the reusable numerical property that matters more in numerical reasoning. Therefore, we argue that injecting numerical properties into symbolic placeholders with contextualized representation learning schema can provide a way out of the dilemma in the number representation issue here. In this work, we introduce this idea to the popular pre-training language model (PLM) techniques and build MWP-BERT, an effective contextual number representation PLM. We demonstrate the effectiveness of our MWP-BERT on MWP solving and several MWP-specific understanding tasks on both English and Chinese benchmarks.

## introduction recent works in math word problem (mwp) solving @xcite @xcite @xcite @xcite @xcite @xcite arrange the pipeline into a sequenceto-sequence framework. in brief, they use deep representation and gradient optimization as well as symbolic constraints to discover discrete symbolic combinations of operators and variants. fundamentally, mwp solving system aims to perform symbolic reasoning by searching through a combinatorial solution space given the text description text: some workers are producing 660 clothes. it has been 5 days and 75 clothes are produced per day. but they have to finish all clothes in 3 more days. how many clothes should be processed per day from now? ## related works math word problems solving. there exist two major types of mwp, equation set mwp @xcite and arithmetic mwp @xcite . this work focuses on arithmetic mwp, which is usually paired with one unknown variable. along the path of the mwp solver's development, the pioneer studies use traditional rule-based methods, machine learning methods and statistical methods @xcite @xcite . afterwards, inspired by the development of sequence-to-sequence (seq2seq) models, mwp solving has been formulated as a neurosymbolic reasoning pipeline of translating language descriptions to mathematical equations with encoder-decoder framework @xcite @xcite @xcite . by fusing hard constraints into decoder @xcite @xcite @xcite , mwp solvers achieve much better performance then. several works propose to utilize multi-stage frameworks @xcite @xcite to make more robust solvers. also, several new works made attempts to improve mwp solver beyond supervised settings @xcite ). among all these previous studies, the most relevant ones to our work can be categorized into two groups. first, it has been noted that number values and mathematical constraints play a significant role in supporting numerical reasoning. @xcite proposed several number value features to enhance encoder and @xcite designed new auxiliary tasks to enhance neural mwp solvers. compared with their work, we first introduce pretraining language model (plm) and concentrate on representation learning to resolve numerical understanding challenges. second, regarding the usage of pre-training techniques for mwp solving, @xcite introduced bart-based @xcite ) mwp solver and incorporated specialized multi-task training for obtaining more effective pre-training seq2seq models for mwp. compared with them, our work focuses on the number representation learning issue of mwp and achieves a more flexible pre-training representation module for mwp solving, which can be applied in various mwp related tasks other than solution generation. numeracy-aware pre-training models. number representation has been recognized as one of the main issues in word representation learning. existing methods make use of value, exponent, subword and character methods @xcite to obtain number representations for explicit number values. these methods are known to be less effective in extrapolation cases like testing with numbers not appearing in the training corpus. previous related works @xcite @xcite ) mainly focus on shallow numerical reasoning tasks shown in drop dataset @xcite , which usually serves as a benchmark for evaluating numerical machine reading comprehension (num-mrc) performance. compared with mwp solving, num-mrc's main focus is laid on extracting answer spans from a paragraph, which are more fault-tolerant with no needs to predict number tokens. besides, their solution generation tasks only contain simple computations like addition/subtraction and there are only integers in drop. more exactly, several research efforts have been made to deal with this kind of math-related reading comprehension task by synthesizing new training examples @xcite , incorporating special modules considering the numerical operation @xcite and designing specific tokenization strategies @xcite . since mwp solving requires further consideration of the complex composition of reasoning logics in mwp text, the symbolic placeholder is more effective in mwp solving. thus, instead of dealing with explicit number values, our work focuses on improving representation for symbolic placeholders by injecting numerical properties in a probabilistic way. ## experiments we present several empirical results with octopus evaluation settings @xcite to prove the superiority of mwp-bert and mwp-roberta solver. in section 4.1, we illustrate the application of both solvers in the generation scenario, mwp solving, by fine-tuning them with a specific decoder @xcite . next, we present mwp probing tasks in section 4.3 to evaluate the capability of mwp-bert and mwp-roberta on "understanding" or "capturing the meanings" of mwps. finally, results and analysis about ablation study are illustrated in section 4.4. implementation details. we pre-train our model on 4 nvidia tesla v100 graphic cards and fine-tune on 1 card. the model was pre-trained for 50 epochs (2 days) and fine-tuned for 80 epochs (1 day) with a batch size of 32. adam optimizer (kingma and ba, 2014) is applied with an initial learning rate of 5e-5, which would be halved every 30 epochs. dropout rate of 0.5 is set during training to prevent over-fitting. during testing, we use 5-beam search to get reasonable solutions. the hyper-parameters setting of our bert and roberta is 12 layers of depth, 12 heads of attention and 768 dimensions of hidden features. for the chinese pre-training model, we use an upgrade patch of chinese bert and roberta which are pre-trained with the whole word masking (wwm) foot_0 @xcite . for the english pre-training models, we use the official source on this website foot_1 . our code and data have been open-sourced on github foot_2 . ## conclusion we propose mwp-bert, an mwp-specific plm model with 8 pre-training objectives to solve the number representation issue in mwp. also, a new dataset ape-clean is curated by filtering out unsolvable problems from ape210k, and the filtered mwps are useful for self-and weakly-supervised pre-training. experimental results show the superiority of our proposed mwp-bert across various downstream tasks on generation and understanding. in terms of the most representative task mwp solving, our approach achieves the highest accuracy, and firstly beats human performance. better numerical understanding ability is also demonstrated in the probing evaluation. we believe that our study can serve as a useful pre-trained pipeline and a strong baseline in the mwp community.