Tree-Based Representation and Generation of Natural and Mathematical Language (2023)

Abstract: Mathematical language in scientific communications and educational scenarios is important yet relatively understudied compared to natural languages. Recent works on mathematical language focus either on representing stand-alone mathematical expressions, especially in their natural tree format, or mathematical reasoning in pre-trained natural language models. Existing works on jointly modeling and generating natural and mathematical languages simply treat mathematical expressions as text, without accounting for the rigid structural properties of mathematical expressions. In this paper, we propose a series of modifications to existing language models to jointly represent and generate text and math: representing mathematical expressions as sequences of node tokens in their operator tree format, using math symbol and tree position embeddings to preserve the semantic and structural properties of mathematical expressions, and using a constrained decoding method to generate mathematically valid expressions. We ground our modifications in GPT-2, resulting in a model MathGPT, and demonstrate that it outperforms baselines on mathematical expression generation tasks.

## introduction a part of human communication is performed in rigorous mathematical language rather than more flexible natural language, which often occurs in scenarios such as scientific communication and education. while pre-trained large language models such as bert @xcite and gpt-3 @xcite have enjoyed many successes in representing and generating natural language, there is a need for models that are effective in representing and generating principled mathematical language as well. while existing work focuses on various aspects of mathematical language representation or generation, combining mathematical language with the aforementioned models for natural language remains a challenging problem. mathematical and natural language are fundamentally different in many ways. while natural language consists of large sets of words and phrases that often have their meaning grounded in context, mathematical language consists of different symbols: a small set of mathematical operators with precise meaning, variables, and numbers that exist in a continuous space. furthermore, mathematical language follows rules that are much more strict and rigorous than natural language. for example, the multiplication operation acts on exactly two operands, while an integral operates on a single operand but with upper and lower limit arguments. operands are either variables, numbers, or the result of applying other operations. given its hierarchical nature, mathematical language is naturally represented with operator trees (opts), which are directed tree graphs where non-leaf nodes are operators and leaf nodes are variables or numbers @xcite . opts are effective at capturing both the semantic and structural properties of mathematical expressions. existing work primarily focuses on two separate approaches to modeling mathematical language: representation and mathematical reasoning. a line of work focuses on learning meaningful representations of mathematical expressions (often formulas), such as @xcite ; @xcite , motivated by the task of retrieving similar expressions, which is especially relevant in information search and retrieval. although these methods produce dense representations of expressions in their natural tree form, they cannot be directly connected to natural language. some works employ bert-like models to jointly represent natural and mathematical language @xcite @xcite . however, these methods are not well suited for generation tasks. another line of work focuses on mathematical reasoning, motivated by the task of mathematical problem solving that is especially relevant in educa-tional applications. these works treat problem solving as a sequence-to-sequence task @xcite and have found success on solving word problems @xcite . state-of-the-art methods use pre-trained large language models @xcite and can even generate meaningful step-by-step solutions @xcite . however, these works do not take the principled structure of math into account and treat mathematical expressions as sequences of math latex tokens in the same way as text tokens @xcite . ## methodology we now detail our proposed modifications grounded in our model mathgpt, visualized in figure 1 . first, we detail how natural and mathematical language are represented jointly by the model. second, we detail how we provide the model with token-level tree position information, followed by how we represent math token embeddings via a learnable transformation on text token embeddings. third, we detail our rules for constrained decoding and tree structure inference at test time. ## experimental setup we now detail a series of experiments to validate the effectiveness of mathgpt. we perform pre-training on a large corpus of math-focused wikipedia articles and then use the model on various generative downstream tasks involving both natural and mathematical languages. ## experimental results we now detail quantitative experimental results to validate the effectiveness of mathgpt in jointly modeling natural and mathematical languages. ## qualitative analysis we now qualitatively investigate how mathgpt represents the semantic and structural aspects of mathematical language differently than gpt-2 finetuned on the same math wikipedia articles. ## related work representations of mathematical language existing work on studying the representations of mathematical language is mainly motivated by information retrieval, i.e., retrieving a semantically and/or structurally relevant mathematical expression (often formula) given a query @xcite . both representations based on expert-crafted rules @xcite and those learned from large-scale scientific formula data @xcite have been shown to be highly effective at this task. however, most of these works do not consider the important textual context around expressions. several recent works jointly model text and math: topiceq (yasunaga and lafferty, 2019) learns topic keywords associated with expressions, mathsum @xcite generates headlines for mathematical discussion forum posts, and one of the mathbert models @xcite learns how to grade students' open-ended math responses. however, none of these works leverage the tree structure of math expressions. another mathbert model @xcite and a recent work @xcite jointly represent textual tokens and expressions in their tree format. however, neither is naturally suited for generation tasks. ## conclusions and future work in this paper, we proposed a series of modifications to common language models to represent and generate text and math jointly. we applied these modifications to gpt-2, resulting in the mathgpt model, which excels at capturing the semantic and structural properties of mathematical expressions in generative tasks. there are many avenues of future work, including i) develop representations of expressions that are invariant under structural transformations that do not change their semantic meaning, ii) conduct human evaluation to validate the quality of the generated mathematical expressions in multiple aspects, iii) develop similarity metrics for mathematical language (like codebleu (ren et al., 2020) for code) and validate them with human evaluation, and iv) explore mathgpt's usage in math education such as word problem generation @xcite and open-ended answer prediction @xcite .