Structural information in mathematical formulas for exercise difficulty prediction: a comparison of NLP representations (2022)

Abstract: To tailor a learning system to the student’s level and needs, we must consider the characteristics of the learning content, such as its difficulty. While natural language processing allows us to represent text efficiently, the meaningful representation of mathematical formulas in an educational context is still understudied. This paper adopts structural embeddings as a possible way to bridge this gap. Our experiments validate the approach using publicly available datasets to show that incorporating syntactic information can improve performance in predicting the exercise difficulty.

## introduction online learning platforms aim to provide personalised tutoring at scale using data-driven personalisation @xcite . a key component of a personalised system is a recommendation algorithm that suggests the next learning activity. to ensure that the recommendation is tailored to the student's level and learning needs, not only should the student's ability level model be considered, but also the learning content characteristics, such as its difficulty. learning content can contain multiple media types (images, text, formulas), each of which must be converted to a numeric format compatible with machine learning models. while natural language processing (nlp) and computer vision allow us to efficiently represent texts and images, the meaningful representation of formulas in an educational context is still understudied. this paper proposes a method for representing mathematical expressions (considered as a form of text) based on an structural embeddings and investigates its effectiveness in predicting exercise difficulty. ## related work research directions in mathematics can be broadly categorised into three branches: generation, assess-ment and solving. in each task, we need to represent a mathematical exercise that may include a text description, a formula, and a picture. the majority of works in this area focuses on word problems that can be represented as bag-of-words @xcite (optionally with binary indicators of whether the word is a mathematical term). such a representation allows the use of rich semantic taggers that provide additional information about lexical units, such as their degree of concreteness or associated emotions @xcite . however, semantic taggers are usually developed for general language use rather than for a specialised domain such as mathematics with its large variety of special characters. previous work accommodated this by manually introducing additional mathematical symbols to be parsed @xcite or by considering entire mathematical expressions as tokens, an approach called bag-of-expressions @xcite . however, such an approach ignores the order of mathematical symbols. a possible extension is to use n-grams to represent chunks of symbols, thus preserving partial information about their order @xcite . its downside is that it is limited by the chosen length of the n-gram and thus cannot fully account for deeply nested expressions. in short, these approaches to representing content tend to be simplistic and do not allow syntactic or semantic information to be fully exploited. therefore, previous research suggests that hierarchical representations should be used to capture deep features and generate higher quality content features @xcite , an observation that motivates our study. in natural language processing, the next level of representation after n-grams is a parse tree of a sentence. it captures syntactic information by representing words as nodes connected by syntactic dependencies: for example, an adjective used as a modifier of a noun. similar to a natural lan- figure 1 : left: a parse (constituency) tree for the sentence "he is kind" (simplified). right: a parse tree for the mathematical expression 1 + 2 * x. leaf nodes are in bold. this paper compares possible ways to encode syntactic information in mathematical exercises and adopts the structural embeddings approach to represent mathematical formulas @xcite . each sentence is represented as a constituency parse tree. in such a tree for a natural language utterance, the non-terminal nodes correspond to grammatical relations (for example, np stands for "noun phrase"), while the leaf nodes contain words. each word is then represented as a sequence of nodes in a parse tree from its leaf to the root of the sentence. for example, if we want to represent the word "kind" in the sentence "he is kind", we construct the parse tree (figure 1 ) and obtain the corresponding syntactic sequence [s, vp, a]. a representation obtained this way captures hierarchical information while facilitating the use of standard neural network models, e.g., an lstm (long-short term memory) @xcite . we refer the reader to the original paper for architectural details and intuition behind them. generally speaking, it encodes a variable-length syntactic sequence into a fixed-length vector representation -the syntactic-semantic embeddings -and the final hidden state serves as input to a decision-making model. ## data the experiments are conducted on two datasets: a recently released math @xcite foot_0 and a synthethic deepmind mathematics @xcite foot_1 , one of the largest publicly available datasets for mathematics in educational data mining. the deepmind dataset consists of question and answer pairs, and each pair has a label corresponding to the difficult level, easy or hard. an advantage of this dataset is its clear and unified formatting: exercise often have a consistent phrasing that mostly differs on a formula step, which allows us to have a clearer comparison of how effective different formula representations are. for our experiments, we used a subset of 37 244 problems, covering a broad range of topics: algebra, arithmetic, calculus, comparison, numbers, polynomials. 18 608 problems are of high level and 18 636 are of low. word descriptions are in english and formulas are not specifically separated but can be extracted using regex-based approach due to limited variability of the rest of the sequence. the math dataset contains 12 498 problems from mathematics competitions in the us. problems are labelled with five levels of difficulty (1001, 2242, 2723, 2904 and 3628 problems, respectively) and cover the following topics: algebra, counting & probability, geometry, intermediate algebra, number theory, prealgebra, precalculus. word descriptions are in english and formulas are written in latex and defined by $ operators. ## experiments neural networks were implemented in tensorflow with @xcite and trained on google colab pro gpus. we used early stopping, monitoring validation loss with the patience of 3 epochs. ## conclusion & future work we proposed an adaptation of an nlp technique @xcite from the field of machine comprehension to the area of mathematical educational data mining. we enrich the content representation by parsing mathematical formulas into syntax trees and embedding them with neural networks. our experiments validate the approach using publicly available datasets and show that incorporating syntactic information can improve performance in predicting the difficulty of an exercise. these results suggest that the method may be of interest for personalised learning solutions. we hypothesise that the advantage of structural embeddings will be more evident for more advanced tasks. therefore, as a next step, we plan to apply our approach to more complex state exams. data have been collected and ocr-processed for initial experiments, and we intend to make the dataset publicly available. our future research will also focus on predicting the similarity of mathematical formulas by comparing their syntax trees.