Towards Grounding of Formulae (2020)

Abstract: A large amount of scientific knowledge is represented within mixed forms of natural language texts and mathematical formulae. Therefore, a collaboration of natural language processing and formula analyses, so-called mathematical language processing, is necessary to enable computers to understand and retrieve information from the documents. However, as we will show in this project, a mathematical notation can change its meaning even within the scope of a single paragraph. This flexibility makes it difficult to extract the exact meaning of a mathematical formula. In this project, we will propose a new task direction for grounding mathematical formulae. Particularly, we are addressing the widespread misconception of various research projects in mathematical information retrieval, which presume that mathematical notations have a fixed meaning within a single document. We manually annotated a long scientific paper to illustrate the task concept. Our high inter-annotator agreement shows that the task is well understood for humans. Our results indicate that it is worthwhile to grow the techniques for the proposed task to contribute to the further progress of mathematical language processing.

## introduction in modern research, scientific progress is often solely shared in digital form. especially in technical research fields, such as in science, technology, engineering, and mathematics (stem), it is a crucial aspect to access data and new results in a quick and uniform way. nevertheless, mathematical formulae, which transport essential information in scientific documents, often remain semantically unutilized in large databases such as digital library of mathematical functions (dlmf) @xcite and the pre-print archive arxiv.org (hereafter re-ferred to as arxiv). by applying math information retrieval (mathir) techniques based on natural language processing (nlp), we are able to utilize this extra knowledge of mathematical formulae to build scientific knowledge bases (kbs) @xcite , improve mathematical search engines @xcite @xcite , or even convert entire scientific papers into executable formats @xcite . formulae often express key ideas in scientific documents. consequently, working with stem documents requires to grasp the meaning and intention of the respective formulae. in other words, grounding of formulae is crucial for processing stem documents and developing its applications. however, the grounding is not a trivial process because of the flexibility of mathematical notation and the impreciseness of natural languages. first, generally, formulae in documents are not independent content that can be understood separately from surrounding texts. for this reason, some initiative projects, e.g., the mathematical language processing (mlp) project @xcite , the mathcat project @xcite , and the part-of-math (pom) tagger @xcite , have been undertaken to integrate nlp techniques into formula analysis. we also follow this mlp direction. second, there is a necessity of disambiguation of mathematical notation because a letter or symbol in formulae is not used in a constant single meaning in a document @xcite . the usage of notation is highly-flexible and, as we will show in this paper, a notation can be used for several meanings even in a paragraph. this notation flexibility is not a problem for tasks that targeting short fragments of text, e.g., the arqmath task @xcite for question posts from a question answering website. however, it is necessary to perform the grounding in consider-ation of the flexibility for processing longer stem literature. it is difficult to perform such a grounding in the scopes of existing tasks or mlp tools because many of the tools and approaches are not capable of carrying multiple meanings for a single symbol in a document @xcite @xcite @xcite . therefore, we will propose a new task direction of grounding of formulae (figure 1 ) in order to take the flexibility into account. in short, the grounding is procedures to identify smallest groups of letters and symbols in formulae, i.e., math words, that independently refer to a mathematical concept and associate the math words with a corresponding text description or an entry in an external kb. for example in figure 1 , the first and the third y are parts of math words y(•) (where • represents an arbitrary argument) and associated with a description of a function while the second y is an independent math word and associated with a description for an output vector. in our grounding, instead of directly associating each math word to a text description, we put an intermediate procedure: making groups each of which consists of math words referring to an identical mathematical concept. for instance in figure 1 , the first and the third y belong to a group because they both refer to the same function, while the second one is in another group because it refers to a vector. it is notable that the grounding is similar to some established tasks, namely coreference resolution @xcite and named entity recognition @xcite . in this work, we checked the feasibility of the proposing task direction for the grounding. for this purpose, we made a long annotated scientific paper in which all formulae are annotated with math word spans and text descriptions of the corresponding mathematical concepts. the math words in the annotated paper which refer to the same mathematical concept are tied together in a group. we did the annotation for an entire paper rather than small fragments of texts to disclose the flexibility of math word usage. through the analysis of our annotated paper, we revealed that the meanings of math words can be changed even within the scope of a single paragraph in actual stem literature. in addition, we did the annotation by multiple human annotators and calculated the inner-annotator agreements so that to confirm that our task design can be well-understood, at least for human beings, and can be performed without individual differences. ## related work a few tasks that are similar to our grounding of formulae have been proposed in the community of mathir. the ntcir project run several shared tasks in the past @xcite @xcite . the task designs of these shared tasks focuses on applications, namely searching and question-answering, while our grounding focuses rather basic parts and is regarded as preprocess for numerous mlp tasks including formulae searching, question-answering, and conversions to formal languages. notably, the ntcir-10 mathir test collection @xcite , which is a specific dataset for their shared task, contains textual descriptions for formulae components, and thus the data are similar to our annotated paper. the dataset includes 10 papers chosen from the arxm-liv dataset using a manually annotated description for each formula element. for instance, in the dataset, a formula log(𝑥) is annotated with a description like "a function that computes the natural logarithm of the value 𝑥". though their purpose is close to ours, we annotated not only descriptions but also a few pieces of additional information, i.e., affix types and group information (what concept the word refer to). in the terms of linguistics, these two can be regarded respectively as word spans and coreference information. additionally, we did the annotation for a longer document than their target papers with coherency. we were especially interested in longer documents so that we can analyze how diverse meanings of mathematical concepts can be. the variable typing task @xcite is also closely relevant to our goal. their task is simply associating mathematical type (technical terms referring to mathematical concepts) to each variable in stem documents. for example, for a sentence let 𝑃 be a parabolic subgroup of gl(𝑛) with levi decomposition 𝑃 = 𝑀 𝑁, where 𝑁 is the unipotent radical. @xcite they assigned the "parabolic subgroup" and "unipotent radical" respectively to variables 𝑃 and 𝑁 as their mathematical types. based on arxmliv, they introduced their own dataset, which includes 33,524 labeled variables in 7,803 sentences. their work re- the result of running the machine learning algorithm can be expressed as a function y(x) which takes a new digit image x as input and that generates an output vector y, encoded in the same way as the target vectors. the precise form of the function y(x) is determined during the training phase. (p. 2, prml) mathematical concepts • function y(•) • output vector y figure 1 : the grounding of formulae. it is a 3-step procedure: (1) detecting the span of math words-𝑦(•) is a word for the first and third y, and the sole y is a word for the second y in the above quote (where • can be arbitrary argument), (2) making groups based on the corresponding mathematical description-the first and third y are elements of a group because they both refer to the same function, while the second one is an element of another group because it refers to a vector, (3) associating each group to text description or external knowledge. ## grounding of formulae in order to describe the grounding of formulae precisely, we introduce a few linguistic terms for formulae, roughly borrowing from morphology in natural languages @xcite : every math word has a corresponding mathematical concept such as the sign function, the set of all natural numbers, and (real) intervals. in the actual data and applications, the math words are associated with textual descriptions. those descriptions can be taken either from the surrounding text of the formulae or from an external kb. though some combinations of words, notably an entire formula, also refer to a mathematical concept, we stick to the scope of math words for the grounding. the process of combining the math words and interpreting the constituted concepts will be a subsequent task to our grounding. it is also notable that the concept of math word is close to mathematical objects of interests (mois) @xcite , sub expressions in formulae that can be identified as 'important' components, but not exactly the same. while mois can contain other mois, math words are minimal groups of morphemes that can refer to mathematical concepts, so they naturally cannot contain other math words. with the terms we introduced in the above, we can describe the proposing grounding of formulae as the following 3-step procedure that simulates the processes of understanding formulae by human beings (figure 1 ): with this step, we will obtain the groups each of which consists of math words appearances used in the same meaning. in the example of figure 1 , the first and the third y belongs to a group, each math word of which refers to a function, while the second y is in another group, each of which refers to a vector. although the grounding is a fundamental step for mlp, the processes, even merely recognizing the word spans, are not easy for computers because of various linguistic phenomena in formulae. the following paragraphs show the two most important phenomena: both make the grounding highly challenging. in short, because of these phenomena, the grounding demands disambiguation of math words. for disambiguation, integration of nlp and analyses for formulae are inescapable. more phenomena are discussed in detail elsewhere in the literature @xcite . all quotes presented in this section are from a textbook pattern recognition and machine learning (prml) @xcite . integration of formulae and texts formulae in scientific documents are generally deeply integrated into narrative texts and inseparable from natural languages. for instance, in the following sentence, the equation is the passive subject in the grammar of english. for the case of a single real-value variable 𝑥, the gaussian distribution is defined by from the viewpoint of the formula, meanings of some math words (𝑥, 𝜇, and 𝜎 2 ) are described in the surrounding natural language texts. moreover, natural language texts or their fragments can appear in formulae, as shown in the following: given this definition of likelihood, we can state beyes' theorem in words posterior ∝ likelihood × prior where all of these quantities are viewed as functions of w. (p. 22, prml) the natural language parts of the above are just nouns, but they can be sentences, e.g., {𝑛 ∈ n | 𝑛 is even}. because of this integration phenomenon, the grounding cannot be done merely by analyzing formulae. for faithful grounding, one must look into natural language texts to observe information such as the contexts to which formulae belong and to definitions, descriptions, or assumptions for identifiers in formulae. ambiguity various ambiguities arise in formulae. token-level analyses are responsible for disambiguation @xcite . different from natural languages, the meanings of words in formulae invariably depend on the context. it is common in stem documents that a notation has multiple meanings even in a single document. for example, in figure 1 , a letter y is used in two meanings. herein, in the first appearance, y is a function, but it is a vector in the second appearance. we human beings can see that y is used in different meanings in these two appearances by the apposition nouns (in this example, the words "function" and "vector" immediately before the formulae) and the usages of the notations. in the first formula, the morpheme y is not used alone, but with affixes; it constitutes a word y(•). however, the second y is used as a single word with no affixes. there are also syntactic ambiguities in formulae. for instance, a formula 𝑓 (𝑎 + 𝑏) can be interpreted in two ways: it means 𝑓 × (𝑎 + 𝑏) if 𝑓 is a variable, or applying the value of 𝑎 + 𝑏 to a function 𝑓 . ambiguities of these types can be resolved by disambiguating the meanings of math words (the meanings of 𝑓 and the parentheses in this case). ## manual annotation for the grounding we performed manual annotation for the proposed grounding task so that to show the feasibility of the task with sufficient reproducibility. moreover, language resources are indispensable to automate the task to contribute to the development of mlp tools. the annotated document for this work should play an initiative role to develop larger language resources for developing and evaluating the grounding technology. collections of natural languages and corresponding formal expressions are not enough for this purpose, because it does not directly provide token-level information for each formula in natural language texts. it is still difficult to extract such information from simple parallel translations. in order to reveal the flexibility of the math word usage, we annotated an entire long paper rather than small fragments of multiple texts. as we described, the grounding can be regarded as a threestep process. corresponding to the first two steps of the grounding, the reference data should also have two types of information for each math morpheme in documents: (1) which word the morpheme belongs to and (2) which mathematical concept the word is referring to. we annotated these two pieces of information coherently for all formulae in a document. we also annotated text description by an annotator for each group, that is the information corresponding to the result of step 3, but it is not literally extracted from the paper itself nor from external knowledge bases. these textual description are rather for the convenience for the annotators. the classification-based annotation made it possible to evaluate the quality of annotated data. moreover, the annotation is expected to be beneficial for ## future work we made a long annotated paper and show that the flexibility of the mathematical notation is high in actual stem literature. moreover, we could check the feasibility of our task direction of the grounding of formulae. the number of annotated papers for the grounding needs to be increased because a single paper is naturally biased. however, the entirely manual annotation costs too much to enhance the size of the resource in the same way. therefore, we will work on partial automation of the process first. with the combination of the partial automatic method of the grounding and manual annotation by humans, we will be able to efficiently enlarge the resource. furthermore, we will develop an entirely automated grounding method, including the third step, i.e., the part of associating the groups with text descriptions or external knowledge, for various mlp applications.