Unbiased Math Word Problems Benchmark for Mitigating Solving Bias (2022)

Abstract: In this paper, we revisit the solving bias when evaluating models on current Math Word Problem (MWP) benchmarks. However, current solvers exist solving bias which consists of data bias and learning bias due to biased dataset and improper training strategy. Our experiments verify MWP solvers are easy to be biased by the biased training datasets which do not cover diverse questions for each problem narrative of all MWPs, thus a solver can only learn shallow heuristics rather than deep semantics for understanding problems. Besides, an MWP can be naturally solved by multiple equivalent equations while current datasets take only one of the equivalent equations as ground truth, forcing the model to match the labeled ground truth and ignoring other equivalent equations. Here, we first introduce a novel MWP dataset named UnbiasedMWP which is constructed by varying the grounded expressions in our collected data and annotating them with corresponding multiple new questions manually. Then, to further mitigate learning bias, we propose a Dynamic Target Selection (DTS) Strategy to dynamically select more suitable target expressions according to the longest prefix match between the current model output and candidate equivalent equations which are obtained by applying commutative law during training. The results show that our UnbiasedMWP has significantly fewer biases than its original data and other datasets, posing a promising benchmark for fairly evaluating the solvers’ reasoning skills rather than matching nearest neighbors. And the solvers trained with our DTS achieve higher accuracies on multiple MWP benchmarks. The source code is available at https://github.com/yangzhch6/UnbiasedMWP .

## introduction math word problem (mwp) solving is a longstanding challenging task in natural language pro-question: how many packets are instant noodles less than candy? cessing (nlp) and has attracted lots of attention recently @xcite @xcite @xcite @xcite @xcite @xcite . an automatic mwp solver should not only understand the problem's semantic information but also reason the grounded mathematical relationships implicit in the problem, so that it can transform natural language into solution expression. more recently, deep learning methods @xcite @xcite @xcite have made great progress in mwp solving and achieved impressive results on several popular benchmarks, such as math23k @xcite and mawps @xcite . however, there exists some severe possible solving bias in these benchmarks, consisting of data bias and learning bias. here, the data bias is introduced since the training dataset does not fully cover diverse questions for each problem narrative of all mwps, leading to the situation that a solver only learns shallow heuristics rather than deep seman-tics for understanding problems. besides, even the question of an mwp is deleted, a solver still can solve it correctly, as shown in figure 1(a ). on the other hand, an mwp can be solved by multiple equivalent equations while current popular datasets only take one of the equivalent equations as the ground truth output for each sample, forcing the model to learn the labeled ground truth and ignore other equivalent equations which may be more suitable for a solver to learn, leading to learning bias during training. as shown in figure 1(b) , if a solver may generate an answer-corrected expression that is different from ground-truth expression, it will be thought an error and the loss between the answer-corrected expression and the ground-truth expression will be back-propagated to the solver during training, leading to over-correct the solver. this learning bias makes it harder to learn to reason out answer-corrected expressions. to mitigate the solver bias for pushing advanced models to learn underlying reasoning skills rather than solely matching nearest results, we first build a novel mwp dataset, unbiasedmwp, to cover diverse questions for each problem narrative of all mwps. it is constructed by varying the grounded expressions in our collected data and annotating them with corresponding new questions manually, thus mitigating data bias. then, to mitigate the learning bias, we propose a dynamic target selection (dts) strategy to dynamically select the most suitable target expression by applying the longest prefix match between the current model output and candidate equivalent equations obtained by applying commutative law during training. our experimental result shows that our unbiasedmwp has significantly fewer biases than its original data and other datasets, and the solvers equipped with our equivalent expression matching loss can achieve higher accuracy on multiple mwp benchmarks such as math23k and our unbiasedmwp. our main contributions are in two folds: ## unbiasedmwp dataset in this section, we introduce the construction procedure of our unbiasedmwp dataset. based on the newly-collected raw data, we design a pipeline for pre-processing and rewriting questions according to formula variations, which is strictly performed by the annotators to obtain unbiased data. ## dynamic target selection strategy during the common mwp training procedure, only one expression is used as ground truth while the equivalent expressions are ignored. consider the following case: the ground truth label is "(n 1 * n 0) -n 0" while the model output is "(n 0 * n 1) -n 0". although they are mathematical equivalent, the model output is judged to be incorrect. therefore, models are prone to be biased during training. to address this issue, we generate the equivalent expressions of the original ground truth expression and then select an equivalent expression matching the longest prefix with the current model output as target expression in the training procedure. ## conclusion in this paper, we revisit the solving bias in mwp. to mitigate the data bias caused by lacking question diversity, we construct a data set called un-biasedmwp by variating the expressions in newcollected data. the experimental results illustrate that the solver trained on unbiasedmwp is more robust than on our collected data. to mitigate the learning bias caused by loss overcorrect with taking only one ground-truth, we proposed a strategy to generate the equivalent expressions and select the longest prefix with the current model output during training, called dynamic target selection (dts). experimental results show that our dts