Investigating Math Word Problems using Pretrained Multilingual Language Models (2022)

Abstract: In this paper, we revisit math word problems (MWPs) from the cross-lingual and multilingual perspective. We construct our MWP solvers over pretrained multilingual language models using the sequence-to-sequence model with copy mechanism. We compare how the MWP solvers perform in cross-lingual and multilingual scenarios. To facilitate the comparison of cross-lingual performance, we first adapt the large-scale English dataset MathQA as a counterpart of the Chinese dataset Math23K. Then we extend several English datasets to bilingual datasets through machine translation plus human annotation. Our experiments show that the MWP solvers may not be transferred to a different language even if the target expressions share the same numerical constants and operator set. However, it can be better generalized if problem types exist on both source language and target language.

## introduction how to use machine learning and nlp techniques to solve math word problems (mwps) has attracted much attention in recent years @xcite @xcite @xcite @xcite . given a math problem expressed in human language, a mwp solver typically first converts the input sequence of words to an expression tree consisting of math operators and numerical values, and then invokes an executor (such as the eval function in python) to execute the expression tree to obtain the final numerical answer. figure 1 shows an example math word problem, the correct expression tree, and the final answer. despite the relatively simple syntax of these expression trees, building mwp solvers is not a trivial task, and researchers have proposed various methods to tackle the different challenges of this problem such as statistical methods @xcite problem: a chef needs to cook 9 potatoes. he has already cooked 7. if each potato takes 3 minutes to cook, how long will it take him to cook the rest? et @xcite , parsing-based methods @xcite and generation-based methods @xcite . however, an aspect that has been largely overlooked is cross-lingual and multilingual mwp solving, i.e., whether a mwp solver trained on one human language can still work on another human language, or whether a mwp solver trained on multiple human languages together is more effective than a solver trained on only one language. we believe this is an interesting aspect to study for the following reasons. first, in cognitive science, people have long studied the relationship between humans' numerical processing abilities and language abilities, and found that on the one hand, the two are largely independent @xcite , but on the other hand, "acquiring and mastering symbolic representations of exact quantities critically depends on language and instruction" @xcite . it is therefore also intriguing to study whether machines separately acquire arithmetic and language abilities. second, with pre-trained large-scale multilingual language models such as mbert @xcite and xlm-r (conneau et al., 2020), which presumably project different human languages into a common embedding space, we have seen some success in cross-lingual nlp tasks such as xnli @xcite and mlqa @xcite in both zero-shot and few-shot settings @xcite . it is therefore reasonable to expect that for mwp solving, there is the possibility of transferring machine's capability of mwp solving from one language to another by leveraging these pre-trained multilingual language models. in this paper, we conduct an empirical study to understand to what extent mwp solvers can work in cross-lingual and multilingual settings. specifically, we ask the following questions: (1) crosslingual setting: given a model trained with monolingual dataset, can the model solve mwps over another language? (2) multilingual setting: can combining datasets of different languages further boost the performance for each language? (3) can we identify some critical factors that may affect the results in (1) and (2)? in order to empirically answer the questions above, we need multilingual mwp datasets, which are limited currently. we first use large scale datasets like math23k @xcite and mathqa @xcite as monolingual mwps resource and further adapt mathqa to have the same operator set and expression style with math23k. to better evaluate the models with parallel corpus, we extend some existing mwp datasets by translating them from english into chinese. we then conduct three sets of experiments on the constructed datasets. we find that: (1) a cross-lingual mwp solver finetuned on one language cannot work on a second language, even if they are sharing the same decoding vocabulary, (2) a multilingual mwp solver may not boost performance for all the training languages but can improve those problems of similar types if one training language is close to the evaluation language, (3) combining (1) (2), we think for multilingual mwp solvers, despite language similarity, the performance relies heavily on domain similarity (problem types). our work makes the following contributions: (1) to the best of our knowledge, we are the first to study cross-lingual and multilingual mwp solving, and we empirically demonstrate that crosslingual mwp solving is still difficult, but multilingual mwp solving is to some extent effective. ## related work solving math word problems (mwps) has been attracting researchers since the emergence of artificial intelligence. student @xcite is a rule-based math word problem solver which contains a pipeline that consists of heuristics for pattern transformation. many researchers start with the fundamental problem types like addition and subtraction @xcite or those that have only one single operator @xcite . @xcite look at problems that require multisteps using two or more operators. the question types of mwps are also expanding. rather than focusing on problems that need only one variable, @xcite propose a dataset alg514 which includes problems with a system of equations. with the development of deep learning, there has been a demand for large-scale datasets with more variations. dolphin18k @xcite is a large-scale dataset that is more than 9 times of the size of previous ones, and contains many more problem types. math23k @xcite ) contains math word problems for elementary school students in chinese language and is crawled from multiple online education websites. mathqa @xcite is a new large-scale, diverse dataset of 37k multiple-choice math word problems in english and each problem is annotated with an executable formula using a new operationbased representation language. hmwp @xcite ) contains three types of mwps: arithmetic word problems, equations set problems, and nonlinear equation problems. various approaches have been proposed to solve mwps. template-based approaches @xcite @xcite are widely adopted as numbers appeared in the expressions are usually sparse in the representation space and the expressions may fall into the same category. more recently, the community is also paying more attention to train a math solver by fine-tuning pretrained language models. for example, ept @xcite adopts albert @xcite as the encoder for its sequence-to-sequence module. the monolingual performance gains achieved recently have not been evaluated from cross-lingual and cross-domain perspectives. therefore, we decide to revisit mwps using current sota pretrained multilingual language models to construct a competitive math solver and conduct experiments over various bilingual evaluations. ## cross-lingual and multilingual mwp solvers in this work, as we are focusing on the cross-lingual and multilingual properties of mwps, we need to train separate mwp solvers using different datasets. our cross-lingual mwp solver will be trained using one language but evaluated using another. our multilingual mwp solver can be trained on all languages available and evaluated separately. to suffice these goals, it would be better if the problems in different languages have comparable properties. since we are using pretrained multilingual language models as the sequence embedder of the encoder, all the languages can be projected into a shared representation. however, the candidate datasets also need to share a common operator set and numerical constants to make the decoding process consistent. but some of the categories from mathqa do not exist on math23k or one of the operators is not in our permitted set. therefore, we need to adapt mathqa as a counterpart of math23k sharing the same decoding vocabulary, including operators and constants. ## template-based contrastive training math word problems can be categorized by expression templates if we replace numerical values of expressions with a special token. such templates have been adopted for supervision in other math solver approaches like @xcite and @xcite . different from these methods, we don't use templates directly for supervision but make an assumption that problems sharing the same template are closer with each other from the point view of arithmetics, regardless of the surface forms of languages and descriptions. to make use of this assumption, we introduce inter-language template-base contrastive training into our training process. specifically, we first group math word problems based on their templates. during training, we pair each problem with a random sample from a different language sharing the same template. as the representation learned by the encoder in section a is m, we use its maxpooling with normalization as the latent representation for each problem and its positive sample, denoted as z and z + respectively. then, we conduct a batch-level contrastive training similar to simclr (chen et al., 2020) and use the nt-xent loss (the normalized temperature-scaled cross entropy loss) as the fol-lowing: our contrastive learning approach differs with that of @xcite in the following ways: (1) our method is focusing on cross-lingual setting that each pair of examples come from different languages, (2) we use batch-level contrastive training in consist with simclr. there are also other works making use of latent representations of math word problems to enhance generalization ability of math solvers. for example, liang and zhang (2021) designed a teacher module to make the latent vector to match the correct solution rather than its variations. ## experiments 6.1 experiment setup evaluation metrics: the model is expected to be a math problem solver, so the generated expressions should be executable by a specific compiler and executor. during evaluation, each problem is counted as solved if the absolute error rate for the executed value and the target value is lower than a predefined threshold. in our experiments, we choose 1e metric is the accuracy of solved problem against all the problems. methods to be compared: we empirically compare the following cross-lingual: (1) mbert-zh is using original multilingual bert @xcite but trained over math23k only; other experiment settings: we choose to use multi-lingual bert (mbert) @xcite for cross-lingual training. we train our models using one nvidia 2080ti and a batch size of 160. the learning rate is set to 3e -5 with a scheduler supporting polynomial decay. the training lasts for at most 150 epochs and will stop after 30 epochs if no improvement is observed. foot_0 ## conclusion in this paper, we revisit the math word problems using a generation-based method constructed over pretrained multilingual models. to assist analysis of cross-lingual properties of math solvers, we adopt two large-scale monolingual datasets and further adapts mathqa into the same annotation framework with math23k. we also reuse earlier smaller datasets and upgrade them into bilingual datasets by machine translation and manual checking. our experiments show that the mwp solvers may not be transferred to a different language even if the target expressions have the same operator set and constants. but for both cross-lingual and multilingual cases, it can be better generalized if problem types exist on both source language and target language. problems considered to be easy by humans may still be hard for a math solver trained over the same language but from a different domain. this tells us that for math word problem solvers, it might be beneficial to consider balancing different question types and permitted operators during training.