Neural Machine Translation for Mathematical Formulae (2023)

Abstract: We tackle the problem of neural machine translation of mathematical formulae between ambiguous presentation languages and unambiguous content languages. Compared to neural machine translation on natural language, mathematical formulae have a much smaller vocabulary and much longer sequences of symbols, while their translation requires extreme precision to satisfy mathematical information needs. In this work, we perform the tasks of translating from LaTeX to Mathematica as well as from LaTeX to semantic LaTeX. While recurrent, recursive, and transformer networks struggle with preserving all contained information, we find that convolutional sequence-to-sequence networks achieve 95.1% and 90.7% exact matches, respectively.

## introduction mathematical notations consist of symbolic representations of mathematical concepts. for the purpose of displaying them, most mathematical formulae are denoted in presentation languages (pl) @xcite such as l a t e x @xcite . however, for computer-interpretation of formulae, machine-readable and unambiguous content languages (cl) such as mathematica or semantic l a t e x are necessary. thus, this work tackles the problem of neural machine translation between pls and cls as a crucial step toward machineinterpretation of mathematics found in academic and technical documents. in the following, we will illustrate the ambiguities of representational languages. those ambiguities range from a symbol having different meanings over notational conventions that change over time to a meaning having multiple symbols. consider the ambiguous mathematical expression (x) n . while pochhammer @xcite himself used (x) n for the binomial coefficient x n , for mathematicians in the subject area of special functions, more precisely hypergeometric series, (x) n usually denotes the pochhammer symbol, which is defined for natural numbers as (x) n := x @xmath0. this work uses l a t e x as pl and mathematica as well as semantic l a t e x as cls. mathematica is one of the most popular computer algebra systems (cass); we use mathematica's standard notation (inputform) as a cl (from now on, for simplicity, referred to as mathematica.) semantic l a t e x is a set of l a t e x macros that allow an unambiguous mathematical notation within l a t e x @xcite and which has been developed at the national institute for standards and technology (nist) by the digital library of mathematical functions (dlmf) and the digital repository of mathematical formulae (drmf). in l a t e x, the pochhammer symbol (x) n is simply denoted as (x)_n. in semantic l a t e x, it is denoted as \pochhammersym{x}{n} and compiled to l a t e x as {\left(x\right)_{n}}. in mathematica, it is denoted as @xcite and can be exported to l a t e x as (x)_n. to display them, it is generally possible to translate formulae from cls to pls, e.g., mathematica has the functionality to export to l a t e x, and semantic l a t e x is translated into l a t e x as a step of compilation. however, the reverse translation from pl to cl is ambiguous because semantic information is lost when translating into a pl. mathematical formulae are generally similar to natural language @xcite . however, mathematical formulae are often much longer than natural language sentences. as an example v k q figure 1 : schema of recurrent (top left), recursive (bottom left), transformer (middle), and convolutional sequenceto-sequence (right) neural networks. by applying convolutional sequence-tosequence networks, this work achieves an exact match accuracy of 95.1% for a translation from l a t e x to mathematica as well as an accuracy of 90.7% for a translation from l a t e x to semantic l a t e x. in contrast, the import function of the mathematica software achieves an exact match accuracy of 2.7%. on all measured metrics, our model outperforms export / import round trips using mathematica. ## methods we briefly discuss recurrent, recursive, and transformer architectures and then discuss convolutional sequence-to-sequence networks in detail because they showed, by far, the best results. recurrent neural networks showed the worst performance. our experiments used long-short-term-memory (lstm) recurrent networks but did not achieve any exact matches on long equations of the semantic l a t e x data set. this is not surprising as recurrent neural networks generally have poor performance regarding long-term relationships spanning over hundreds of tokens @xcite . for our data sets, the longest shortest path in the neural network easily exceeds 2 000 blocks. note that the exact match accuracy on such long equations produces successful responses only for a very well-performing model; getting most symbols correct does not constitute an exact match. for a definition of exact matches, see section 5.1. recursive neural networks showed slightly better performance of up to 4.4% exact match accuracy when translating from l a t e x into semantic l a t e x. this can be attributed to the fact that the longest path inside a recursive neural network is significantly shorter than in a recurrent neural network (as the longest shortest path in a tree can be much shorter than the longest shortest path in a sequence.) further, an additional traversal into postfix notation allows for an omission of most braces/parentheses, which (on the semantic l a t e x data set) reduced the required amount of tokens per formula by about 20 -40%. similar to the recurrent networks, we also used lstms for the recursive networks. note that training recursive neural networks is hard because they cannot easily be batched if the topology of the trees differs from sample to sample, which it does for equations. transformer neural networks significantly outperform previous architectures. in our best experiments, we achieved performances of up to 50% exact matches on the mathematical functions site data set. this leap in performance can be attributed to the elaborate multi-headed attention mechanism underlying the transformer model. because we experimented simultaneously with the convolutional sequence-to-sequence architecture and the transformer architecture, and the performance of convolutional networks was by a large margin better (> 90%) than the best performance on transformer neural networks, we decided to set the focus of this work on convolutional networks only. we note that in natural language translation, transformer models typically outperform convolutional neural networks @xcite @xcite . ## limitations in this work, we evaluated neural networks on the task of back-translating mathematical formulae from the pl l a t e x to semantic cls. for this purpose, we explored various types of neural networks and found that convolutional neural networks per-form best. moreover, we observed that the perplexity of the translation of mathematical formulae behaves differently from the perplexity of the translation between natural languages. our evaluation shows that our model outperforms the mathematica software on the task of interpreting l a t e x produced by mathematica while inferring the semantic information from the context within the formula. a general limitation of neural networks is that trained models inherit biases from training data. for a successful formula translation, this means that the set of symbols, as well as the style in which the formulae are written, has to be present in the training data. ## conclusion in this work, we have shown that neural networks, specifically convolutional sequence-to-sequence networks, can handle even long mathematical formulae with high precision. given an appropriate data set, we believe that it is possible to train a reliable formula translation system for real-world applications. we hope to inspire the research community to apply convolutional neural networks rather than transformer networks to tasks that operate on mathematical representations @xcite @xcite @xcite @xcite @xcite . we think that convolutional networks could also improve program-to-program translation as source code has strong similarities to digital mathematical notations-after all, l a t e x and mathematica are programming languages.