Mapping probability word problems to executable representations (2021)

Abstract: While solving math word problems automatically has received considerable attention in the NLP community, few works have addressed probability word problems specifically. In this paper, we employ and analyse various neural models for answering such word problems. In a two-step approach, the problem text is first mapped to a formal representation in a declarative language using a sequence-to-sequence model, and then the resulting representation is executed using a probabilistic programming system to provide the answer. Our best performing model incorporates general-domain contextualised word representations that were finetuned using transfer learning on another in-domain dataset. We also apply end-to-end models to this task, which bring out the importance of the two-step approach in obtaining correct solutions to probability problems.

## introduction solving math word problems automatically is an active area of research in natural language processing. it poses interesting challenges in extracting relevant entities and quantities from a concise textual narrative, and reasoning about the relationships between them to answer the question posed in the text. while several approaches for different areas of mathematics have been proposed in the past, the majority have focused on arithmetic and algebraic problems where the mathematical representation is specified as a simple equation system that can be used with a standard solver @xcite @xcite @xcite @xcite @xcite . this paper focuses on a class of word problems that have not received much attention: those about probability that may be found, for example, in introductory textbooks for discrete mathematics. ## approaches to solving probability word problems our ultimate objective is to address the following: given: a natural language description of a word problem involving computing a probability. ## qualitative analysis we base the error analysis on the investigation of the predicates in the gold representation from the test set that are not found in the predicted one. we classify an output with respect to the first type of predicate that does not find a correspondence in the predicted representation, according to the following order: group, size, given, take, observe, probability. we choose such or-der because each predicate type relies on the information provided by the previous ones. in fact, errors usually result in incorrect predictions for the following types in the order as well. we take as source of our analysis the predictions of the best-performing model, i.e. mathqa-bert (finetuned). of the 214 predictions, 75 are correct and 139 present a missing statement on one of the aforementioned levels. broadly speaking, two types of errors stand out: i) those involving confusion about what kind of modifier or boolean operator to choose, and ii) those involving extracting the right numeric argument for a predicate (e.g. how many items to sample using a take statement). we now analyse the errors in more detail. object sets and their cardinality there are 6 group statements that were not predicted (e.g. error a1 in the appendix), 4 of them on problems with two different groups g 1 and g 2 with a fine-grained composition of the form given(exactly(rel(...,g1)): the network predicts only one group and links the composition of both sets to a single group. there are 10 problems with size statements not found in the corresponding prediction: 5 because the predicted size was wrong (e.g. error a2) and 2 because the statement was missing at all. in both cases the following given(exactly...) statements do not add up to the correct size. ## conclusion we have investigated the use of neural sequenceto-sequence models to generate intermediate representations for solving probability word problems, and shown the benefit of introducing contextualised word representations together with transfer learning on another dataset of math word problems. our results also suggest that mapping to a problem specification followed by the application of a dedicated solver is preferable to end-to-end modelling where the answer probabilities are predicted directly from the encoded text. the qualitative analysis of results reveals that the extraction of relevant entities and quantities from concise textual descriptions, as well as reasoning about the relationships between them are still challenging, and therefore provide possible directions for future work.