Analogical Math Word Problems Solving with Enhanced Problem-Solution Association (2022)

Abstract: Math word problem (MWP) solving is an important task in question answering which requires human-like reasoning ability. Analogical reasoning has long been used in mathematical education, as it enables students to apply common relational structures of mathematical situations to solve new problems. In this paper, we propose to build a novel MWP solver by leveraging analogical MWPs, which advance the solver’s generalization ability across different kinds of MWPs. The key idea, named analogy identification, is to associate the analogical MWP pairs in a latent space, i.e., encoding an MWP close to another analogical MWP, while leaving away from the non-analogical ones. Moreover, a solution discriminator is integrated into the MWP solver to enhance the association between an MWP and its true solution. The evaluation results verify that our proposed analogical learning strategy promotes the performance of MWP-BERT on Math23k over the state-of-the-art model Generate2Rank, with 5 times fewer parameters in the encoder. We also find that our model has a stronger generalization ability in solving difficult MWPs due to the analogical learning from easy MWPs.

## introduction math word problem (mwp) solving has attracted considerable attention in recent years. currently, mwp solver design focuses on generating an equation towards an unknown quantity, with an input problem description, as shown in figure 1 . building such a successful solver is quite challenging, as it requires mathematical understanding and multistep reasoning abilities to transform the implied logic behind the problem into a mathematical equation composed of operators and quantities. with the emergence of deep learning, mwp has been effectively studied by seq2seq models @xcite and pre-trained language models (plms) @xcite problem 2: li has @xmath04 to buy pencils, and the rest of the money buys exercise books at @xmath110 to buy some stationery. he buys 4 pencils at @xmath21 each. he wants to use the rest of the money to buy pens at $2 each, how many pens can he buy? solution: (10-4*0.5-2*1)/2 problem 1: a rope is 24 meters long, we cut 12 meters to make a long jump rope, and use the rest of ropes to make a short jump ropes with a length of 4 meters. how many short jump ropes can be made? 2021; @xcite , which treat it as a translation task from the natural language description of a problem to its solution in a mathematical equation. in educational science, analogical reasoning @xcite has long been considered as a crucial skill in human problem-solving. in education and training of mathematical reasoning, researchers @xcite interpret it as "a dynamics process which, by enabling us to increase the complexity of ideas we can handle, expands our understanding." for example, the mwp p1 and p2 in figure 1 are in different situations, but both aim to calculate the available quantity of a target object based on known resources, leading to the same solution structure. the mwp p2 and p3 both focus on the knowledge of 'number of @xmath3, while the mwp p3 has more reasoning steps. gaining the skill of generating the solution of p2 to address p1 and p3 is important for students, as well as for automated mwp solvers. a widely accepted defi-nition of analogical reasoning is mapping the process of mathematization between different domains @xcite . we thus propose a strategy of associating the mathematization process of analogical mwp pairs, e.g., p1 and p2, p2 and p3. the key idea is to encode the analogical problems close in the latent representation space, while encoding the non-analogical problems distantly. the introduction of analogical learning in mwp solving can then alleviate the limitations of existing methods such as simple-to-difficult generalization @xcite and topic-to-topic generalization @xcite . while the analogical mwps have close representations, they should be also more associated with their corresponding true solutions than wrong solutions. we thus encode the solution as well, and design a module named solution discrimination to enhance the association of an mwp and its correct solution. the enhancement is powered by generating hard negative samples that are wrong solutions but similar to the correct solution. negative sampling has been proved effective in training strong mwp solvers @xcite . for example, @xcite used negative examples to calculate a contrastive loss and find patterns across mwps, but the negative examples are retrieved from the given training set, which could be bounded by the dataset size and quality. @xcite bridged the problem space and solution space, by building a teacher module to help the solver match problems with their ground truth solutions and distinguish them from randomly generated negative solutions. however, the negative samples in their approach are randomly manipulated variants of the ground truth equation, and are thus limited in their shapes and fail to consider the hardness of negative samples. therefore, we design a gradient-based manipulation method to find hard negative samples (i.e., wrong solutions that are difficult to be distinguished from a correct solution). in the meantime, to make the form of negative samples less bounded by the ground truth, we introduce more randomness during the generation of negative samples. overall, our proposed mwp solver is novel on considering the analogy among mwps. it contains an analogy identification module which aims to transfer mathematization knowledge and skills to generalize the solution to analogical mwps. moreover, a solution discrimination module is incorpo-rated to enable the solver to bridge problems and their ground truth solutions. our proposed solver is a flexible framework and can have any mwp encoder-decoder plugged in. the experimental results show that training representative baselines (e.g., gts @xcite and mwp-bert @xcite ) by our analogical strategy can further improve their accuracy. comparing to the current best solver on math23k, generate&rank @xcite , we make mwp-bert achieve higher accuracy but with 5 times fewer encoder parameters. ablation studies, qualitative analyses and case studies also demonstrate the effectiveness of our designed solver framework, and verify its generalization ability in solving difficult mwps due to the analogical learning from easy mwps. ## experimental results in this section, we firstly introduce the used datasets, evaluation metrics and baselines in this paper. then we conduct an accuracy comparison between our solvers and all baseline methods. next, our ablation studies show the contribution from different modules. we also analyze the solver accuracy under different solution lengths to evaluate generalization ability. visualization of different groups of mwps shows that our solver has better mwp representation ability. case studies can be found in appendix. algorithm 1 training pipeline input: problem x, solution y , analogy identification module θ, solution discrimination module γ, encoder-decoder module δ parameter: loss weights λ 1 , λ 2 output: well-trained θ, γ and δ 1: for x, y in the training set do 2: x pos , x neg ← sample positive/negative problems from the training set by analogy identification. y neg ← replace y i with all possible tokens and get the negative solution set. train the discriminator γ with cross-entropy loss. train the analogy identification module (i.e., mlp) γ and encoder-decoder module δ with a joint loss: @xcite is the most commonly used chinese dataset in mwp solving. it contains 23,162 problems with 21,162 training problems, 1,000 validation problems and 1,000 testing problems. we use the value accuracy as the evaluation metric, which checks whether the equation solution given by the model leads to the ground truth value. mathqa mathqa @xcite is an english mathematical problems dataset at gre level. the original mathqa dataset is annotated in a different way from math23k with many pre-defined operations. also, some problems in the dataset suffer from the low-quality issue. many efforts @xcite @xcite have been paid to clean and filter the mathqa dataset. in our experiment, we follow the latest version @xcite of mathqa dataset where 4 arithmetic operators {+, -, * , /} are included in this subset. after data filtering, the training set contains 16191 training mwps and 1605 testing samples. we also use the value accuracy as our evaluation metric for this dataset. ## conclusion we propose a novel analogical training pipeline for math word problem solvers, considering the generalization among analogical mwps and the association between ground truth solutions and problem descriptions. in this way, problems that need similar solving skills are grouped together to support analogical learning, and solvers are trained to focus on ground truth solutions. the comparative study shows that our method with mwp-bert outperforms other baselines on math23k and mathqa, and is much lighter (with fewer parameters). it can also solve more difficult problems due to the analogical reasoning. we believe our work would facilitate the mwp research community and inspire more studies about the analogy in mathematical question answering.