Textual Enhanced Contrastive Learning for Solving Math Word Problems (2022)

Abstract: Solving math word problems is the task that analyses the relation of quantities e and requires an accurate understanding of contextual natural language information. Recent studies show that current models rely on shallow heuristics to predict solutions and could be easily misled by small textual perturbations. To address this problem, we propose a Textual Enhanced Contrastive Learning framework, which enforces the models to distinguish semantically similar examples while holding different mathematical logic. We adopt a self-supervised manner strategy to enrich examples with subtle textual variance by textual reordering or problem re-construction. We then retrieve the hardest to differentiate samples from both equation and textual perspectives and guide the model to learn their representations. Experimental results show that our method achieves state-of-the-art on both widely used benchmark datasets and also exquisitely designed challenge datasets in English and Chinese.

## introduction solving math word problems (mwps) is the task of automatically performing logical inference and generating a mathematical solution from a natural language described math problem. solving mwps is a challenging task that cannot rely on shallow keyword matching but requires a comprehensive understanding of contextual information. for example, as shown in figure 1 , while the first problem shares high token-level overlapping with the third problem, the underlying mathematical logic is different. while on the other hand, the first and second problems have very low similarity at the textual level, while the equation solution is the same. the challenge of the task is that the underlying problem: dave bought boxes of chocolate candy and gave to his little brother. if each box has pieces inside it, how many pieces did dave still have? ## methodology we use contrastive learning to obtain text features with high differentiation of small perturbations, so that for each anchor data point @xmath0, e), where t stands for the text and e stands for the equation, we construct a pair of examples positive data point p + = (t + , e + ) and negative data point p -= (t -, e -), and then use contrastive learning loss to map the representation of p and p + closer and vice versa. the pipeline of the triplet pairs retrieval is shown in figure 2 . we first construct a candidate pool, which consists of supervised training data {p i } and augmented selfsupervised data {p aug i } as shown in the blue part of figure 2 . the self-supervised data is generated by two methods, reversed operation based data augmentation (roda) and question reordering (qr), which is explained in section 3.1. then we perform two-step retrieval to retrieve the triplet pairs as described in section 3.2. we first use an equation-based retrieval strategy to extract positive candidate set { p + } and negative candidate set { p -}, and then further introduce textual information by choosing one example from the candidate set via a text-based retrieval strategy. finally, we train the mwp solving model that maps t to e by considering both the contrastive learning and solution equation generation objective, as described in section 3.3. ## conclusion in this paper, we propose a textual enhanced contrastive learning framework, which leverages both supervised and self-supervised supervision to help the model understand contextual information and bridge subtle textual variance to mathematical logic. we use two novel task-specific data augmentation methods to enrich the candidate pool with examples with minor textual variance for contrastive learning triplet pair retrieval. we design a two-stage retrieval method to find hard example triplet pairs with both equation and textual information and investigate various retrieval strategies. experimental results show that our method gained improvement on both benchmark datasets and challenge datasets in english and chinese. we also conduct visualization for representation distribution on different equations and also on a case study, which shows our method can benefit the representation learning. with the combination of data augmentation, our method still improves the performance and achieves sota results on math23k dataset.