Modeling Mathematical Notation Semantics in Academic Papers (2021)

Abstract: Natural language models often fall short when understanding and generating mathematical notation. What is not clear is whether these shortcomings are due to fundamental limitations of the models, or the absence of appropriate tasks. In this paper, we explore the extent to which natural language models can learn semantics between mathematical notation and their surrounding text. We propose two notation prediction tasks, and train a model that selectively masks notation tokens and encodes left and/or right sentences as context. Compared to baseline models trained by masked language modeling, our method achieved significantly better performance at the two tasks, showing that this approach is a good first step towards modeling mathematical texts. However, the current models rarely predict unseen symbols correctly, and token-level predictions are more accurate than symbol-level predictions, indicating more work is needed to represent structural patterns. Based on the results, we suggest future works toward modeling mathematical texts.

## introduction with the enormous growth of academic publishing, there is increasing interest in the area of scholarly document analysis in nlp @xcite . many academic papers use mathematical notation, both in formulas and in describing components of algorithms, as seen in 'α is the learning rate', and '240 × 240 pixel image'. however, despite the great advances in pretrained language models such as bert @xcite , they are still unable to analyze mathematical notation reliably @xcite . similarly, in our experiments ( §4.3), pretrained models show very poor performance (9%), when compared with the traditional n-gram based models (19%). this empirical result tells us that bert's pretraining method barely incorporates the modeling of mathematical notation. although we find that finetuning through masked language modeling on academic text increases the model performance (by 48%), the accuracy is still low for an application. we cast the problem of mathematical notation semantics as one of predicting the right notation given the context -notation prediction has the advantage of simplicity. motivated by an academic paper authoring application @xcite , we propose two new target tasks that can make use of mathematical notation prediction: notation autosuggestions and notation consistency checks. notation auto-suggestions: some symbols are used conventionally in a given domain. for instance, the symbol α is used conventionally in deep learning papers as learning rate. a tool for suggesting notation could learn about conventions from usages across many papers, and make suggestions for appropriate notation given its context. notation consistency checks: when writing technical papers, authors develop the ideas during the process of writing. this can lead to inconsistency in notation. for instance, a concept might be referred to as d to mean delta in equation 1 in the paper, and then later the authors may use d to mean document in equation 3. a notation verification tool could check the consistency of notation usage across the paper and warn the author when the different uses of the symbol might be in conflict. in addition to the tasks, we propose an approach to fine-tuning bert language models to represent mathematical notation with a level of accuracy that begins to approach that needed for these kinds of real-world applications. as shown below, our method achieves top-1 token-level accuracy of 61% and 74% for notation auto-suggestion and notation consistency checks, respectively. 1 our contributions can be summarized as follows: the use of mathematical notation in texts. mathematical notation is integral to academic discourse. the symbols in a single academic paper often number in the hundreds @xcite . a single instance of notation can be simple, as in the case of a single-letter symbol, or complex, as in an equation composed of dozens of symbols. in a given text, the prose and notation are closely related and inter-referential. studies of reading behavior reveal readers diverting their focus from notation to text and vice versa @xcite @xcite . definitions of notation appear close to the notation itself; @xcite estimated that as many of 70% of symbols are defined in the same paragraph that the symbol is introduced. this close relationship between prose and notation suggests that language models may be able to select notation to suit the context it appears in. modeling mathematical notation. neural networks have been used widely to predict textual tokens from their context, with simple models such as word2vec @xcite and high-capacity transformers @xcite . however, could similar techniques be used to predict mathematical notation given the textual context it appears in? recent research has explored this possibility in various tasks such as type inference in mathematical statements @xcite by using transformer architecture @xcite , topic-sensitive equation generation @xcite , superscript disambiguation @xcite with recurrent neural networks, and mathehttp://github.com/hwiyeoljo/mathpredictor matical information retrieval (greiner-petter et al., 2020) using word2vec algorithms. these results show that the recent advanced models could be used for modeling notation. a complementary line of research takes notation as input, and attempts to predict text to describe the notation. in recent work, @xcite retrieved a relevant paragraph to a given query, @xcite and @xcite extracted symbol descriptions from text, and @xcite detected symbol descriptions in math equations. in contrast to prior work, our model takes text as input and predicts appropriate mathematical notation. ## proposed method: mathpredictor we design a method to learn the meaning of math notation grounded in the surrounding text. this requires good representations of standard text, mathematical notations, and their combination. to types examples #-uniq letter n, m, sha, model, loss, x .. 16k number 218, 00, 4k, 2k, 90., 2cm .. 234 op&symbol α, θ , ≤, ×, arccos, %, ∃ .. 271 latex macros \top, \text, \mathcal \quad, \bf, \rm, \underline, \em .. 562 our model takes raw latex files as input, which have some markedly different characteristics than general natural language text input. in this section we define the notations that our system recognizes ( §3.1), and how it is tokenized ( §3.2). then we describe our model mathpredictor and its training and inference procedure ( §3.3). ## experiments and evaluation this section evaluates how well our model performs at mathematical notation prediction compared to several strong baselines. we first describe the evaluation dataset ( §4.1), the baseline models ( §4.2), and the notation prediction tasks ( §4.3). we then investigate the following research questions: r1 ( §4.4): how well does our model predict when compared to the baselines for the two prediction tasks? r2 ( §4.5): does the model simply memorize notations in context or does it learn domainconventional patterns from other papers? r3 ( §4.6): which types of notations is the model most able to predict? r4 ( §4.7): how well does the model perform when evaluated at the sentence level? r5 ( §4.8): how well does the model perform at the document-level (qualitatively)? ## discussion mathpredictor as an application. our proposed method showed reasonable performance in top-5 accuracy (71.3% and 80.0% for each task), which is strong for a novel task, but most likely not good enough for a real-world application. however, when we sub-sample 10 times more than the main experiment, the performance on the tasks is improved on the same test set by +10% accuracy: 70.9% for top-1 accuracy and 81.6% for top-5 accuracy on auto-suggestion and 83.5% for top-1 accuracy and 89.0% for top-5 accuracy on consistency checks. these results are promising for mathpredictor to be utilized as an application. the results suggest that current models memorize the meanings rather than generalize over them. although we showed the possibility of modeling mathematical notations, most of the results indicates that the models are not able to predict tokens which have not been presented before (in ## conclusion in this paper, we propose two novel notation prediction tasks to evaluate mathematical notation semantics in academic paper writing. we then present a fine-tuned bert particularly optimized on these tasks, which outperforms other baselines. our analysis shows that the model can be thought of as a way to encode knowledge about the usages of mathematical notations in specific domains, although it does not seem to generalize beyond the notation within the text it is exposed to. we see this as a first step toward more powerful analysis tools that can one day act as a method to help authors of mathematical texts select and refine their notation.