World Models for Math Story Problems (2023)

Abstract: Solving math story problems is a complex task for students and NLP models alike, requiring them to understand the world as described in the story and reason over it to compute an answer. Recent years have seen impressive performance on automatically solving these problems with large pre-trained language models and innovative techniques to prompt them. However, it remains unclear if these models possess accurate representations of mathematical concepts. This leads to lack of interpretability and trustworthiness which impedes their usefulness in various applications. In this paper, we consolidate previous work on categorizing and representing math story problems and develop MathWorld, which is a graph-based semantic formalism specific for the domain of math story problems. With MathWorld, we can assign world models to math story problems which represent the situations and actions introduced in the text and their mathematical relationships. We combine math story problems from several existing datasets and annotate a corpus of 1,019 problems and 3,204 logical forms with MathWorld. Using this data, we demonstrate the following use cases of MathWorld: (1) prompting language models with synthetically generated question-answer pairs to probe their reasoning and world modeling abilities, and (2) generating new problems by using the world models as a design space.

## introduction math story problems (msps) are short narrative texts that describe a dynamic situation in the world consisting of entities, actions and states, followed by a quantitative question about the world, as displayed in fig. 1 . the task of automatically solving msps has received much research attention ## mathworld in this section, we present our world model formalism mathworld. we formalize an msp as a sequence of n sentences @xmath0, denoted g. 1 we refer to the nodes of the graph as containers ( § 3.1) and the edges of the graph as relations ( § 3.2). each container and relation is labelled with a set of properties. one such property is the quantity, which may be either an explicit number mentioned in text or a variable representing an unknown number. the containers and relations along with their properties specify the equations induced by the msp. in addition, each g is associated with a reference variable r, which points to the variable in g that holds the correct answer to the question as stated in q. we consider each s to be associated with some structure (g, r). we say that g is faithful if it represents the semantics of the problem text according to the framework of mathworld. further, g is complete if r can be solved with the equations induced by g. a complete world model is correct if, when evaluated, r gives the correct answer to the problem. see fig. 1 for an example of a world model. in order to allow for incremental parsing, we segment the world models into sentence-level logical forms m i , @xmath1, . . . , n. the logical form is a sequence that represents the containers and/or relations associated with the corresponding sentence. 2 we can convert (m 1 , . . . , m n ) to a world model graph, and vice versa. the two representations are nearly equivalent, with the exception of a few caveats (see app. f for details). there is no bound on the problem length and, by extension, the number of logical forms. mathworld is thus able to represent problems of any arbitrary number of reasoning steps. the assignment of logical forms may be ambiguous in the sense that there may be multiple faithful logical forms for a given sentence (discussed in app. b). we consider subgraphs g i , for sentence i, of the final graph g. a subgraph g i corresponds to the logical forms up to sentence i, i.e., (m 1 , . . . , m i ) → g i . we refer to the subgraph for some sentence index i as the state of i. as an example of how world models are built incrementally with states, consider fig. 1 . the first sentence maps to the container for label will holding the entity money of quantity 83 with unit dollar. the second sentence provides information on an update to will's possessed money, a transfer relation ( § 3.2.1). finally, the question sentence introduces rate information, a rate relation ( § 3.2.2), between money and toys. in the next sections, we describe the details of containers and relations in depth. ## data collection in order to study how models are able to answer msps, convert them to logical form, perform world modeling, and reason mathematically to find the answer, we require a diverse dataset of labeled msps that spans all concepts covered by mathworld. to ensure diversity and wide variety in the examples, we collect them from numerous sources: we randomly sample a subset from each of these three datasets, 9 and annotate them with world models. we obtain 1, 019 msps, which corresponds to 3, 204 logical forms, which we partition into 80/20 train/test splits. we performed an agreement analysis of 125 overlapping msps, revealing a high agreement rate considering the complexity of the annotation task. concretely, 61 out of these 125 were strongly equivalent ( § 3.3) across annotators, and 107 were weakly equivalent ( § 3.3). many of the only weakly equivalent annotations were due to ambiguity in the properties (app. b.1), and almost half of the 18 non-agreed problems were due to ambiguity in relation type (app. b.2). the strong and weak smatch scores were 0.91 and 0.97 respectively. these can be interpreted as approximate upper bounds on the smatch scores achievable by any model, due to the ambiguity in the dataset. many of the annotation errors, also outside of the overlapping set, could be either corrected or discarded ex post. further details on annotation are given in app. e. ## applications of mathworld in this section we showcase some applications of mathworld: solving ( § 5.1), probing of reasoning ( § 5.2) and generation of new msps ( § 5.3). ## conclusion in this work, we have presented a novel formalism, mathworld, for expressing the semantics of math story problems. we have annotated a math-world corpus consisting of 1, 019 problems and 3, 204 logical forms. a world model derived from mathworld exposes the structure of the reasoning process needed to solve the problem, which benefits several applications as we have demonstrated in § 5. as such, we hope that mathworld will promote use cases beyond just improved msp solving, ranging from automated chain-of-thought prompting to math problem generation.