Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks (2021)

Abstract: Previous math word problem solvers following the encoder-decoder paradigm fail to explicitly incorporate essential math symbolic constraints, leading to unexplainable and unreasonable predictions. Herein, we propose Neural-Symbolic Solver (NS-Solver) to explicitly and seamlessly incorporate different levels of symbolic constraints by auxiliary tasks. Our NS-Solver consists of a problem reader to encode problems, a programmer to generate symbolic equations, and a symbolic executor to obtain answers. Along with target expression supervision, our solver is also optimized via 4 new auxiliary objectives to enforce different symbolic reasoning: a) self-supervised number prediction task predicting both number quantity and number locations; b) commonsense constant prediction task predicting what prior knowledge (e.g. how many legs a chicken has) is required; c) program consistency checker computing the semantic loss between predicted equation and target equation to ensure reasonable equation mapping; d) duality exploiting task exploiting the quasi-duality between symbolic equation generation and problem’s part-of-speech generation to enhance the understanding ability of a solver. Besides, to provide a more realistic and challenging benchmark for developing a universal and scalable solver, we also construct a new largescale MWP benchmark CM17K consisting of 4 kinds of MWPs (arithmetic, one-unknown linear, one-unknown non-linear, equation set) with more than 17K samples. Extensive experiments on Math23K and our CM17k demonstrate the superiority of our NS-Solver compared to state-of-the-art methods.

## introduction deep neural networks have achieved remarkable successes in natural language processing recently. although neural models have demonstrated performance superior to humans on some tasks, e.g. reading comprehension @xcite @xcite , it still lacks the ability of discrete reasoning, resulting in low accuracy on math reasoning. thus, it is hard for pure neural network approaches to tackle the task of solving math word problems (mwps), which requires a model to be capable of natural language understanding and discrete reasoning. mwp solving aims to automatically answer a math word problem by understanding the textual description of the problem and reasoning out the underlying answer. a typical mwp is a short story that describes a partial state of the world and poses a question about an unknown quantity or multiple unknown quantities. to solve an mwp, the relevant quantities need to be identified from the text. furthermore, the correct operators along with their computation order among these quantities need to be determined. therefore, integrating neural networks with symbolic reasoning is crucial for solving mwps. inspired by the recent amazing progress on neural semantic parsing @xcite and reading comprehension @xcite , we address this problem by neural-symbolic computing. recently, many researchers @xcite @xcite @xcite , inspired by an encoder-decoder framework @xcite , apply neural networks to solve mwps by learning the mapping function between problems and their corresponding equations, and achieve remarkable successes. the encoder uses a neural network to represent a problem as a real-valued vector, and the decoder uses another neural network to generate an equation or expression token by token. the main difference among previous methods is the way to decode expressions or equations. however, they only follow the encoder-decoder paradigm but lacking the ability to explicitly incorporate essential math symbolic constraints (e.g. commonsense constants, formulation regularization), leading to unexplainable and unreasonable predictions. besides, most of them only focus on arithmetic mwps without any unknown, preventing them from generalizing to various types of mwps, such as equation set problems. to address the above issues, we propose a novel neural-symbolic solver (ns-solver), which explicitly and seamlessly incorporates different levels of symbolic constraints by auxiliary learning tasks. our ns-solver consists of three main components, a problem reader to encode the math word problems into vector representations, a programmer to generate the symbolic grounded equations, which are executed to produce answers, and a symbolic executor to obtain final results. in addition to the supervised training objective between generated symbolic grounded equations and groundtruth equations, our solver is also optimized by four novel auxiliary objectives that enforce four levels of problem understanding and symbolic reasoning. first, we apply number prediction task to predict both the number quantity and number location in the problem in a self-supervised manner. second, we deploy commonsense constant prediction task to predict what prior commonsense knowledge (e.g. how many legs a chicken has) is required for our solver. third, we propose program consistency checker to compute the semantic loss between the predicted program and ground-truth equation to ensure reasonable equation mapping. finally, we also propose a novel duality exploiting task that exploits the quasi duality between symbolic grounded equation generation and the problem's part-of-speech generation to enhance the understanding ability of our solver. there are some key advantages of our solution. first of all, the above four auxiliary tasks can produce additional training signals, which improves the data efficiency in training and makes our solver more robust. second, using the predicted constant to constrain the target symbolic table can reduce the search space greatly, which means that our solver can generate correct symbolic grounded equations easier and better. third, the auxiliary tasks have been proven to help reduce the domain gap between seen and unseen mwps @xcite , thus improving the reasoning ability of our solver. besides, beyond the current large-scale highquality mwp benchmark that only includes one type of problems, we also construct a large-scale challenging chinese mwps dataset cm17k, which contains 4 types of mwps (arithmetic mwps, oneunknown linear mwps, one-unknown non-linear mwps, equation set problems) with more than 17k samples, to provide a more realistic and challenging benchmark for developing a universal and scalable math solver. extensive experiments on public math23k and our proposed cm17k demonstrate the superiority of our ns-solver compared to stateof-the-art methods in predicting final results while ensuring intermediate equation rationality. neural-symbolic computing. neural-symbolic computing has greatly promoted the development of semantic parsing. @xcite ; @xcite ; @xcite applied neural sequence-to-sequence and sequence-to-tree models to semantic parsing with full supervision. @xcite have advanced the stateof-the-art in weakly supervised semantic parsing on knowledge graphs and tabular databases. al-problem reader programmer executor consistency checker gt equation tree 𝑥 𝑦 𝑛 1 𝑛 2 math word problem auxiliary tasks … 26 heads and 82 feet.... number location prediction today there are chickens and rabbits … location: 15;18 ℒ 𝐶𝐶𝑃 chicken_legs, 4 rabbit_legs, 2 commonsense constant prediction duality exploration today there are chickens and rabbits in the same cage, with a total of 26 heads and 82 feet. how many chickens and rabbits are there? encoder decoder pos result today/nn there/ex are/vbp chickens/nns and/cc rabbits/nns … problem pos tagging dual constrains ℒ 𝑁𝐿𝑃 ℒ 𝑁𝑄𝑃 number quantity prediction … 26 heads and 82 feet ... quality of numbers: 2 ℒ 𝑃𝐶𝐶 number mapping 𝑛 1 𝑛 2 26 82 when a problem preprocessed by number mapping and replacement is entered, our problem reader encodes the problem text into context representation. then our programmer generates a tree-structured symbolic grounded program explicitly. finally, a symbolic grounded program will be executed to produce answers by the executor. in our ns-solver, we apply four auxiliary tasks to enhance its problem understanding and symbol reasoning ability for generating better programs. though most of the successes of semantic parsing are limited to structured data sources, it is not expensive for mwps since it is easy to crawl lots of problems with annotated equations and answers. therefore, mwp solving can benefit from supervised neural-symbolic computing. self-supervised learning. self-supervised auxiliary tasks have been widely used in the fields of natural language understanding @xcite . @xcite applied two selfsupervised auxiliary tasks, masked lm and next sentence prediction, to improve the understanding ability of bert by pretraining. albert @xcite introduces sentence-order prediction task to address the ineffectiveness of the next sentence prediction task in bert. @xcite show that self-supervised learning can improve model robustness and uncertainty. dual learning. dual learning, first proposed by @xcite , is a reinforcement training process that jointly trains a primal task and its dual task. then @xcite considered it as a way of supervised learning and designed a probabilistic regularization term to exploit the duality. it has been widely applied in various fields, such as machine translation @xcite , sentiment classification @xcite , question answering @xcite , visual question answering @xcite , machine reading comprehension @xcite , and code generation @xcite . to the best of our knowledge, we are the first to ex-ploit the duality in mwps. different from previous works, we design a quasi dual learning method between symbolic grounded equation generation and problem's part-of-speech generation to enhance the understanding ability by easing the difficulty of generating problems from symbolic equations. ## neural-symbolic solver in this section, we present the design of the proposed ns-solver. its backbone mainly consists of a problem reader that encodes the math word problems into vector representations, a programmer to generate the symbolic grounded programs in prefix order, and a symbolic executor to obtain final results. the overview of our ns-solver is visualized in fig. 1 . we first introduce the backbone of our ns-solver in section 3.1, and then we introduce other auxiliary tasks in section 3.2. ## conclusion in this work, we propose neural-symbolic solver (ns-solver) to explicitly and seamlessly incorporate different levels of symbolic constraints by four auxiliary tasks. our ns-solver consists of a problem reader to encode problems, a programmer to generate a symbolic grounded program, and a symbolic executor to obtain final results. in addition to supervised learning with target expression, our solver is also optimized via four new auxiliary objectives that enforce four levels of symbolic reasoning. besides, we also construct a new dataset cm17k containing 4 types of mwps with more than 17k samples, which provides a more realistic and challenging benchmark for developing a universal and scalable math solver. extensive experiments on math23k and cm17k demonstrate the superiority of our ns-solver compared to state-ofthe-art methods in answer accuracy while ensuring intermediate equation rationality.