P oint to the E xpression: S olving A lgebraic W ord P roblems using the E xpression- P ointer T ransformer M odel (2020)

Abstract: Solving algebraic word problems has recently emerged as an important natural language processing task. To solve algebraic word problems, recent studies suggested neural models that generate solution equations by using ‘Op (operator/operand)’ tokens as a unit of input/output. However, such a neural model suffered two issues: expression fragmentation and operand-context separation. To address each of these two issues, we propose a pure neural model, Expression-Pointer Transformer (EPT), which uses (1) ‘Expression’ token and (2) operand-context pointers when generating solution equations. The performance of the EPT model is tested on three datasets: ALG514, DRAW-1K, and MAWPS. Compared to the state-of-the-art (SoTA) models, the EPT model achieved a comparable performance accuracy in each of the three datasets; 81.3% on ALG514, 59.5% on DRAW-1K, and 84.5% on MAWPS. The contribution of this paper is two-fold; (1) We propose a pure neural model, EPT, which can address the expression fragmentation and the operand-context separation. (2) The fully automatic EPT model, which does not use hand-crafted features, yields comparable performance to existing models using hand-crafted features, and achieves better performance than existing pure neural models by at most 40%.

## introduction solving algebraic word problems has recently become an important research task in that automatically generating solution equations requires understanding natural language. to improve the performance of pure neural models that can solve algebraic word problems, we identified two issues that can be addressed using expression tokens, which are shown in figure 1: (1) expression fragmentation and (2) operandcontext separation. first, the expression fragmentation issue is a segmentation of an expression tree, which represents a computational structure of equations that are used to generate a solution. this issue arises when op, rather than the whole expression tree, is used as an input/output unit of a problem-solving model. for example, as shown in figure 1 (a), using op tokens as an input to a problem-solving model disassembles a tree structure into operators ("×") and operands ("x 1 " and "2"). meanwhile, we propose using the figure 1 : illustration using the word problem in the second issue of operand-context separation is the disconnection between an operand and a number that is associated with the operand. this issue arises when a problem-solving model substitutes a number stated in an algebraic word problem into an abstract symbol for generalization. as shown in figure 1 (b), when using an op token, the number 8 is changed into an abstract symbol 'n 1 '. meanwhile, when using an expression token, the number 8 is not transformed into a symbol. rather a pointer is made to the location where the number 8 occurred in an algebraic word problem. therefore, using such an "operand-context pointer" enables a model to access contextual information about the number directly, as shown in figure 1 (c ); thus, the operand-context separation issue can be addressed. in this paper, we propose a pure neural model called expression-pointer transformer (ept) to address the two issues above. the contribution of this paper is two-fold; in the rest of the paper, we introduce existing approaches to solve algebraic word problems in section 2. next, section 3 introduces our proposed model, ept, and section 4 reports the experimental settings. then in section 5, results of two studies are presented. section 5.1 presents a performance comparison between ept and existing sota models. section 5.2 presents an ablation study examining the effects of expression tokens and applying operand-context pointers. finally, in section 6, a conclusion is presented with possible future directions for our work. ## related work our goal is to design a pure neural model that generates equations using 'expression' tokens to solve algebraic word problems. early attempts for solving algebraic word problems noted the importance of expressions in building models with hand-crafted features @xcite @xcite @xcite . however, recent neural models have only utilized 'op (operator/operand)' tokens @xcite @xcite @xcite , resulting in two issues: (1) the expression fragmentation issue and (2) the operandcontext separation issue. in the remaining section, we present existing methods for tackling each of these two issues. to address the expression fragmentation issue, researchers tried to reflect relational information between operators and operands either by using a two-step procedure or a single step with sequenceto-sequence models. earlier attempts predicted operators and their operands by using a two-step procedure. such early models selected operators first by classifying a predefined template @xcite @xcite , then in the second step, operands were applied to the template selected in the first step. other models selected operands first before constructing expression trees with operators in the second step @xcite . however, such two-step procedures in these early attempts secondly, there were efforts to address the operand-context separation issue. to utilize contextual information of an operand token, researchers built hand-crafted features that capture the semantic content of a word, such as the unit of a given number @xcite @xcite @xcite or dependency relationship between numbers @xcite @xcite . however, devising hand-crafted input features was timeconsuming and required domain expertise. therefore, recent approaches have employed distributed representations and neural models to learn numeric context of operands automatically @xcite @xcite . for example, @xcite used a pointer-generator network that can point to the context of a number in a given math problem. although huang's model can address the operand-context separation issue using pointers, their pure neural model did not yield a comparable performance to the state-of-the-art model using hand-crafted features (44.5% vs. 83.0%). in this paper, we propose that by including additional pointers that utilize the contextual information of operands and neighboring expression tokens, performance of pure neural models can improve. ## ept: expression-pointer transformer figure 2 shows the proposed expression-pointer transformer (ept) foot_0 model, which adopts the encoder-decoder architecture of a transformer model @xcite . the ept utilizes the albert model @xcite , a pretrained language model, as the encoder. the encoder input is tokenized words of the given word problem, and encoder output is the encoder's hidden-state vectors that denote numeric contexts of the given problem. after obtaining the encoder's hidden-state vectors from the albert encoder, the transformer decoder generates 'expression' tokens. the two decoder inputs are expression tokens and the albert encoder's hidden-state vectors, which are used as memories. for the given example problem, the input is a list of 8 expression tokens shown in to produce 'expression' tokens, two components are modified from the vanilla transformer: input vector and output layer. in the following subsections, we explain the two components. ## result and discussion in section 5.1, we first present a comparison study, which examines the ept's performance. next, in section 5.2, we present an ablation study, which analyzes the two main components of ept; expression tokens and operand-context pointers. ## conclusion in this study, we proposed a neural algebraic word problem solver, expression-pointer transformer (ept), and examined its characteristics. we designed ept to address two issues: expression fragmentation and operand-context separation. the ept resolves the expression fragmentation issue by generating 'expression' tokens, which simultaneously generate an operator and required operands. in addition, the ept resolves the operand-context separation issue by applying operand-context pointers. our work is meaningful in that we demonstrated a possibility for alleviating the costly procedure of devising hand-crafted features in the domain of solving algebraic word problems. as future work, we plan to generalize the ept to other datasets, including non-english word problems or non-algebraic domains in math, to extend our model.