Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems (2022)

Abstract: Math Word Problem (MWP) solving needs to discover the quantitative relationships over natural language narratives. Recent work shows that existing models memorize procedures from context and rely on shallow heuristics to solve MWPs. In this paper, we look at this issue and argue that the cause is a lack of overall understanding of MWP patterns. We first investigate how a neural network understands patterns only from semantics, and observe that, if the prototype equations are the same, most problems get closer representations and those representations apart from them or close to other prototypes tend to produce wrong solutions. Inspired by it, we propose a contrastive learning approach, where the neural network perceives the divergence of patterns. We collect contrastive examples by converting the prototype equation into a tree and seeking similar tree structures. The solving model is trained with an auxiliary objective on the collected examples, resulting in the representations of problems with similar prototypes being pulled closer. We conduct experiments on the Chinese dataset Math23k and the English dataset MathQA. Our method greatly improves the performance in monolingual and multilingual settings.

## introduction a math word problem (mwp) is described as a natural language narrative with a math question. the mwp solver is required to generate a solution equation, which can be calculated to get the numerical answer, by understanding the contextual problem description. in teaching, students are encouraged to recognize that mathematics is really about patterns and not merely about numbers @xcite . mathematically excellent students explore patterns, not just memorize procedures @xcite . @xcite mention that existing mwp models @xcite rely on shallow heuristics to generate equations. these models can predict solutions well even if leaving only narratives without questions, which suggests that neural networks learn to solve mwps by memorizing the lexical input like rote learning. thus, existing models get stuck in memorize procedures. we look at this issue and hypothesize it is because they focus on text understanding or equation generation for one problem. the same quantitative relationship corresponds to many problems of different themes and scenarios, but previ-ous methods overlook the outlining and distinction of mwp patterns. in this work, we first investigate how a neural network understands mwp patterns only from semantics. we adopt the widely used encoder-decoder model structure @xcite . @xcite is employed as the semantic encoder, and a tree decoder @xcite ) is adopted to generate equations. we probe the problem representations in bert. the visualization by t-sne (van der @xcite in figure 1 shows that, through the semantic encoder, most representations of problems with the same prototype equation are pulled closer, even if their narratives are semantically different. we also analyze the representations in different bert layers, and the results show the lexical semantics mainly affects the problem-solving in lower layers. besides, for each prototype equation, those problem representations far away from its center representation tend to produce incorrect solutions. inspired by it, we propose a contrastive learning approach that seeks similar prototypes to support model to better understand patterns and perceive the divergence of patterns. when collecting contrastive examples, we follow @xcite to convert the prototype equation to a tree. given an equation tree, the positive examples are retrieved if their trees or subtrees have the same structure, and the negative examples are collected from the rest in terms of the operator types and the size of the tree. the solving model is first jointly optimized by an equation generation loss and a contrastive learning loss on the collected examples, and then, is further trained on the original dataset. while the generation loss empowers the model to memorize procedures from the semantics, the contrastive learning loss brings similar patterns closer and disperses the different patterns apart. we conduct experiments on the chinese dataset math23k @xcite and the english dataset mathqa @xcite in monolingual and multilingual settings. to support constructing multilingual contrastive examples, we follow @xcite to adapt mathqa as the counterpart of math23k. experimental results show that our method achieves consistent gains in monolingual and multilingual settings. in particular, our method allows the model to improve the performance in one language using data in another language, which suggests that mwp patterns are language-independent. furthermore, we verify that, through our contrastive learning, the representations that previously generate wrong solutions get closer to their centers, and several problems are solved well. to summarize, the contributions of this paper include: i) an analysis of the mwp model showing that the semantic encoder understands lexical semantics in lower layers and gathers the prototype equations in higher layers. ii) a contrastive learning approach helping the model to better understand mwp patterns and perceive the divergence of patterns. iii) applications in the multilingual setting suggesting that we can further improve the model performance using data in different languages. ## semantic encoder gathers prototypes in this section, we explore how a neural network understands patterns from semantics. we adopt the encoder-decoder model structure to solve problems, and perform analyses on the problem representations. the observation is that the semantic encoder understands lexical semantics at lower layers and gathers the prototype equations at higher layers. ## contrastive learning in this section, we propose a contrastive learning approach to help the model to perceive the divergence of mwp patterns. one drawback of existing deep learning methods is that they overlook the outlining and distinction of mwp patterns. in contrast, we seek similar prototype equations from various problems to support model to understand patterns, and collect easily confused patterns for model to distinguish. ## experiments we evaluate our method on two widely used datasets @xcite , and demonstrate its effectiveness in monolingual and multilingual settings. ## conclusion in this paper, we find the neural network generates incorrect solutions due to the non-distinction of mwp patterns. to this end, we propose a contrastive learning approach to support the model to perceive divergence of patterns. we seek similar patterns in terms of the equation tree structure and collect easily confused patterns for our model to distinguish. our method outperforms previous baselines on math23k and mathqa in monolingual and multilingual settings.