ATHENA : Mathematical Reasoning with Thought Expansion (2023)

Abstract: Solving math word problems depends on how to articulate the problems, the lens through which models view human linguistic expressions. Real-world settings count on such a method even more due to the diverse practices of the same mathematical operations. Earlier works constrain available thinking processes by limited prediction strategies without considering their significance in acquiring mathematical knowledge. We introduce Attention-based THought Expansion Network Architecture (ATHENA) to tackle the challenges of real-world practices by mimicking human thought expansion mechanisms in the form of neural network propagation. A thought expansion recurrently generates the candidates carrying the thoughts of possible math expressions driven from the previous step and yields reasonable thoughts by selecting the valid pathways to the goal. Our experiments show that ATHENA achieves a new state-of-the-art stage toward the ideal model that is compelling in variant questions even when the informativeness in training examples is restricted.

## introduction math word problem (mwp) solving is one of the fundamental reasoning tasks of answering a mathematical question by understanding a complex, intricate system of human lexical expressions. models' ability to solve a problem depends on a method that articulates the problem, the lens through which they view human lexical expressions. ideal mwp models understand the diverse applications of the same mathematical operations in real-world situations, which require lexically sophisticated. for example, "×" can count all elements equally divided in multiple boxes, but also calculate area from length and width, or tax fee from the tax rate and income. problem 2. the school playground was originally ... how many square meters are increased by the current playground area compared to the original one? problem 1. the school playground was originally [80] meters long and [40] meters wide. later when the school is remodeled, the length is increased by @xcite meters and the width is increased by @xcite meters. how many square meters is the original area? humans learn mathematical knowledge by formulating and understanding underlying principles from seen cases rather than just recognizing the common lexical patterns. models currently face two challenges to reach human-level mathematical understanding: conceptual knowledge to understand the practices of mathematical principles, and procedural knowledge to deductively derive the answer through the principles, which are indispensable for each other in their development and prior approaches mostly adopt the transductionbased models such as sequence-to-sequence @xcite , sequence-totree @xcite or graphto-tree methods @xcite and concentrate on enhancing problem-level encoding @xcite @xcite . these works have limitations in obtaining procedural knowledge due to their prediction strategies that operate in a counter-intuitive order. for instance, sequenceto-tree approaches determine mathematical operations before the operands in the inference steps. recently, @xcite proposed a deductive prediction strategy, but it still entails procedural bias, accepting only one particular reasoning pathway. overall, the prior studies show limitations in learning mathematical procedures, which is significant for achieving successful mathematical skills. as a result, despite their high accuracies on some benchmarks, the current approaches fail to solve variant questions that are simply mutated from already trained examples @xcite . as shown in the cognitive inadequacy of previous models motivates us to propose a new reasoning architecture that maximizes feasible reasoning pathways. we design athena that reasons with thought expansion inspired by the studies of human reasoning @xcite . the key idea is to implement two types of thoughts in the expansion process: thoughts before considering goals formed by conceptual knowledge and goal-directed thoughts yielded by procedural knowledge. figure 1 illustrates an example of thoughts formed via asking different questions (goal) within the same situation (context). because it is tricky for models to answer different questions that share a lexically similar problem context, we state that the two thinking strategies would lead to the right answer by properly utilizing mathematical knowledge. we expand the thoughts by applying conceptual knowledge to obtain candidate thinking pathways and procedural knowledge to evaluate the potential answers. this is how we endow models with mathematical reasoning ability and empirically demonstrate that the model has actually learned the mathematical knowledge. athena puts the aforementioned thinking pro-cess explicitly into neural network propagation. defining the term thought as a representation of each math expression driven from the problem, we shape candidate thoughts and the goal-directed thoughts named reasonable thoughts. the model generates candidate thoughts by applying mathematical operations and yields reasonable thoughts by filtering with solidly updated premises until it meets the appropriate answer. with this recurrent process, we develop a neural model of processing thoughts based on multi-head attention @xcite that effectively carries the subtle feature changes during expansion. our experiments show that the proposed approach is strong at predicting mathematical expressions requiring sophisticated comprehension as shown in ## math word problem math word problem (mwp) solving is the task of answering a mathematical question by understanding natural language descriptions. ## athena attention-based thought expansion network architecture (athena) is an architecture that expands its thoughts to solve mwp. figure 2 illustrates an overall process of athena. athena extracts initial thoughts θ 0 from plm and expands them with inferring through premises until it reaches the final thought. we first clarify what is a thought-a foundational ingredient of our model-and explain the premise and goal vectors that measure the thoughts. thought. a thought is an embedding of a possible math expression derived from quantities in a problem representing the contextual meaning of the expansion expansion infer candidate thoughts context the school playground was originally [80] meters long and [40] meters wide. later when the school is remodeled, the length is increased by [10] meters and the width is increased by [15] meters. question how many square meters are increased by the current playground area compared to the original one? pre-trained language model original length original width increased length increased width [80] premise vector. a premise vector represents previously inferred thoughts to evaluate and filter candidate thoughts in each depth. let p d denote a premise vector for depth d. we set an initial premise vector p 0 with the [cls] token from the problem descriptions. ## experiments we conduct experiments across a comprehensive range of mwp) solving tasks to show that athena outperforms strong baselines in both full datasets and variant versions of the original datasets while being more interpretable in terms of intermediate steps toward the answers. ## conclusion we state that an ideal mwp model needs to be practical in real-world settings that are critical to capture the diverse applications of the same mathematical operations. for this reason, we conclude that athena with thought expansion reaches significant improvements toward the ideal model due to its decent performance on unseen problems or restricted examples to learn.