EPT - X : An Expression-Pointer Transformer model that generates e X planations for numbers (2022)

Abstract: In this paper, we propose a neural model EPT-X (Expression-Pointer Transformer with Explanations), which utilizes natural language explanations to solve an algebraic word problem. To enhance the explainability of the encoding process of a neural model, EPT-X adopts the concepts of plausibility and faithfulness which are drawn from math word problem solving strategies by humans. A plausible explanation is one that includes contextual information for the numbers and variables that appear in a given math word problem. A faithful explanation is one that accurately represents the reasoning process behind the model’s solution equation. The EPT-X model yields an average baseline performance of 69.59% on our PEN dataset and produces explanations with quality that is comparable to human output. The contribution of this work is two-fold. (1) EPT-X model: An explainable neural model that sets a baseline for algebraic word problem solving task, in terms of model’s correctness, plausibility, and faithfulness. (2) New dataset: We release a novel dataset PEN (Problems with Explanations for Numbers), which expands the existing datasets by attaching explanations to each number/variable.

## introduction algebraic word problem solving is a challenging task for understanding natural language. as shown in our novel model, expression-pointer transformer with explanations (ept-x), is inspired by some pedagogical studies about human strategies on understanding an algebraic word problem @xcite @xcite . in classrooms, teachers ask students to make an explanation or a diagram that depicts the role of each number written in the problem. then, students use these explanations for numbers to build a correct equation. that is, understanding a problem produces explanations that satisfy the following two criteria. (1) plausibility: a plausible explanation is one that includes contextual information for the numbers and variables that appear in a given math word problem. especially, as humans recognize each number/variable individually, the explanations should reveal what each number or variable represents in the context of the given problem. (2) faithfulness: a faithful explanation is one that accurately represents the reasoning process behind the model's solution equation @xcite . in other words, they should imply a reason behind selecting operators or operands. to reflect these two criteria in ept-x, we adopt a two-phase architecture: (1) explaining numbers/variables and (2) building solution equations. though @xcite attempted to generate explanations, this work is different from ours in that their model focused on explaining the decoding process. so, they have less explored the above two criteria. in contrast, this paper attempts to explain how the model understands the given word problem by modifying an encoder component of a neural model. as humans successfully solve word problems by explaining their understanding, we expect our ept-x model to achieve a good performance in terms of three criteria: correctness of equations, plausibility of explanations, and faithfulness of explanations. through several analyses, our paper shows the following two contributions: correctness: researchers have attempted to build a model that solves word problems. early attempts used hand-crafted features collected by experts to make a model understand a word problem @xcite @xcite @xcite . although researchers can interpret these models using the features, extending these studies to other datasets is limited as designing features is laborintensive. on the other hand, recent studies have employed neural models @xcite @xcite @xcite ) and achieved answer correctness ranging from 62% to 84%. though their extensibility is better than handcrafted features, it becomes harder to interpret how a neural model understands a word problem. plausibility: to make a neural model that explains its reasoning process, ling et al. ( 2017 ) built a model that outputs both a computation process and a rationale behind the process. though their model generated a natural language phrase that explains a computation step in advance, the model is not enough to meet the plausibility criterion because of two issues. first, it is not guaranteed whether their model explains all numbers and variables required to solve the problem. as they focused more on explaining the model's computation, their model often skips explaining its understanding of numbers and variables stated in a problem. second, it is not confirmed whether their model generates rationale comparable to that of humans. though they measured their quality of rationale using @xcite , the reported score of 27.2 is somewhat low and not compared with any human-level performance. we suspect that this low-quality explanation affected the low correctness of their model: 36.4%. therefore, it is worthwhile to build a new model that fulfills the plausibility criterion. faithfulness: as far as we know, studies on solving algebraic word problems have not measured the faithfulness of a generated explanation. existing studies so far measured the quality of explanations using plausibility only. following jacovi and goldberg (2020), we define faithful explanation as one that accurately represents the reasoning process behind the model's solution equation. humans expect an explanation to be faithful. however, a model can generate an explanation that may not be related to the equations @xcite ; it can generate random plausible sentences independently from the process of generating solution equations. therefore, measuring faithfulness is meaningful in that a highly faithful explanation reflects a solution equation generation process that is expected by human problem solvers. ## the ept-x model the proposed model, expression-pointer transformer with explanations (ept-x) foot_0 , is a variant of expression-pointer transformer (ept; @xcite , which is state-of-the-art correctness model. figure 1 depicts the two phases ept-x model. (1) plausibility: in phase 1, ept-x receives a problem text as an input and generates explanations for each number/variable. the number of variables is also predicted in this phase. (2) faithfulness: in phase 2, ept-x receives both the original problem and the generated explanations as inputs and then builds an equation using ept. to jointly train these two phases, we add up the loss functions for the number of variables, explanations, and equations; all three use smoothed cross-entropy @xcite with α = 0.01. ## the pen dataset we release 'problems with explanations for numbers' (pen) foot_1 , an algebraic word problem dataset with problem texts, equations, and explanations of numbers/variables for each problem to train and evaluate ept-x. as existing datasets for algebraic word problems do not contain explanations, we provided explanations on the existing three benchmark datasets on solving algebraic word problems foot_2 : alg514 @xcite , draw-1k @xcite , and mawps @xcite . the following sections introduce the two stages of building pen: preparation for correcting errors and annotation for collecting explanations. ## experimental setup to verify whether the ept-x model can solve an algebraic problem correctly while generating plausible and faithful explanations, we conduct three types of analyses: model performance analysis, quantitative error analysis, and qualitative output analysis. this section illustrates each analysis and further implementation details. ## result and discussion the result of three analyses reveals that the ept-x can generate an equation correctly based on a plausible and faithful explanation. first, section 6.1 presents the result of the model performance analysis, which shows that ept-x can achieve correctness 5% lower than previous inexplainable models. second, section 6.2 shows the result of error analysis, which reveals that many of ept-x's errors are due to insufficient explanations. and lastly, section 6.3 shows the result of qualitative analysis, which reveals three types of errors found in the explanation generation process of ept-x. ## conclusion this study proposed a novel neural model ept-x, expression pointer transformer with explanations, which generates explanations along with solution equations. the ept-x model was designed to address two criteria of plausibility and faithfulness when generating an explanation. to address plausibility, the model generates explanations for each number/variable in the solution equation separately. and to address faithfulness, the model produces equations based on the information in the generated explanation. in addition to ept-x, we release a new dataset, problem with explanations for numbers (pen), which extends existing three algebraic word problem datasets by augmenting explanations for numbers/variables. using the pen dataset, we conducted three analyses. the model performance analysis revealed that ept-x could produce a correct equation with 69.59% accuracy. the quantitative error analysis showed that the ept-x model could produce a plausible albeit insufficient explanation. lastly, the qualitative output analysis identified three categories of errors made when generating explanations. despite the insufficiency of explanations generated by the ept-x model, our work is significant in that we demonstrated the possibility of generating explanations while solving an algebraic word problem. for future work, we plan to improve the correctness and faithfulness of ept-x to enhance the existing state-of-the-art model.