To be or not to be an Integer? Encoding Variables for Mathematical Text (2022)

Abstract: The application of Natural Language Inference (NLI) methods over large textual corpora can facilitate scientific discovery, reducing the gap between current research and the available large-scale scientific knowledge. However, contemporary NLI models are still limited in interpreting mathematical knowledge written in Natural Language, even though mathematics is an integral part of scientific argumentation for many disciplines. One of the fundamental requirements towards mathematical language understanding, is the creation of models able to meaningfully represent variables. This problem is particularly challenging since the meaning of a variable should be assigned exclusively from its defining type, i.e., the representation of a variable should come from its context. Recent research has formalised the variable typing task, a benchmark for the understanding of abstract mathematical types and variables in a sentence. In this work, we propose VarSlot, a Variable Slot-based approach, which not only delivers state-of-the-art results in the task of variable typing, but is also able to create context-based representations for variables.

## introduction the articulation of mathematical arguments is a fundamental part of scientific reasoning and communication. across many scientific disciplines, expressing relations and inter-dependencies between quantities is at the centre of its argumentation. one of the particular linguistic elements used for such argumentation is variables: they are essential for expressing complex mathematical ideas, allowing scientists to refer to a set of values compactly and rigorously. given the essential nature of variables, models that perform inference over scientific and mathematical text should be able to meaningfully represent and leverage such elements to understand mathematical language and improve downstream inference performance. while there is still debate on a universally accepted definition for variables, their functional aspects within mathematical text are well established. a variable is usually defined as a symbol standing as a referent for a set consisting of at least two elements @xcite . given their nature of abstract and dynamic referents, two aspects make the representation of variables particularly challenging in the field of nlp: (i) the meaning of a variable is exclusively determined by its context @xcite , a variable carries no meaning when considered in isolation, behaving unlike any word in english; (ii) the same variable symbol (e.g., the variable x) can be reused in an unlimited number of sentences and expressions, possibly assuming different meanings and referring to different sets of values while keeping the same name. while a similar problem can also be found for the representation of words (i.e., word sense disambiguation), the scale of ambiguity is more marked, due to the fact that variable names, unlike words, are not grounded a priori to any particular set of concepts. in order to understand mathematical text, the meaning of variables (i.e., their type) needs to be implicitly or explicitly inferred at some point from the text. identifying and qualifying the binding between variables and their types, therefore, is crucial for reasoning with mathematical text, given that any form of inference on a variable is fundamentally constrained by the possible values it can take, and those values are uniquely determined by its type. for example, we can only correctly predict the entailment between two sentences containing variables if we infer their types beforehand. as a benchmark for variable comprehension in mathematical text, the variable typing task @xcite requires finding the connections between variables and their respective types in a sentence. despite its importance for scientific inference, the task is still widely unex-plored. most of the existing work focusing on the representation of mathematical elements, in fact, has been carried out in a non-textual setting, where the mathematical elements are represented independently from any textual content @xcite . while some expressions and formulas are universal enough to be encoded without context (e.g., pythagoras theorem), variables have no meaning in isolation and different symbols can be arbitrarily chosen to indicate variables with the same meaning. therefore, an important principle for designing and evaluating a variable representation is that such a representation should be agnostic to the specific symbols adopted as referents in the text -i.e., the variable names. this is because the variable names contained in a generic passage can be opportunely renamed without altering the sentences' meaning. for instance, a robust variable representation should be able to encode the sentences "let x be an integer" and "let y be an integer" in exactly the same way, if they are inserted in the same context. however, while this characteristic seems to be intrinsic in the nature of variables, it has been largely ignored by current evaluation frameworks, where there is no agnostic way to verify the quality of the generated variable representation independently of their surface form. to move a step forward towards more robust and generalisable representations, this work proposes a new testing and modelling framework based on the property of generalisation to variable renaming. specifically, we define a model to be generalisable to variable renaming if the model's performance does not decrease when renaming variables in the test set with variable names never seen during training. through testing such a property, since the renaming of variables does not alter the context or meaning of the sentences, we are verifying whether the context is correctly moulding the variable encoding and, at the same time, whether the performance is not due to overfitting to the surface form of the text. to address and study generalisation to variable renaming, this work proposes varslot (variable slot), a model for variable typing that represents variables in a surface agnostic way. to achieve a generalisable representation, varslot initialises variables as blank slots and employs a multi-slot mechanism, extending from slot attention @xcite , to iteratively specialise the representation. specifically, varslot leverages self-attention to conform the representation of each variable to its context (i.e., its surrounding words and other variables), where each surrounding element has a different influence on the final representation. experiments adopting varslot to extend sentence embeddings based on transformers @xcite demonstrate not only that the proposed framework is able to achieve state-of-the-art results on the variable typing task, but also, in contrast with previous work, that varslot allows for better generalisation to variable renaming. to the best of our knowledge, this is the first work that focuses on generalisation and robustness for variable representation in mathematical text, providing, at the same time, a critical analysis of large language models (llms) targeting the scientific domain. to summarise, the contributions of this paper are as follows: ## variable typing problem this work follows the same definition of a variable used in @xcite , considering as variables simple expressions composed of a single, possibly scripted, base identifier. the variable typing task requires the assignment between variables in the sentence and its respective types. we use the same setting as @xcite for this task: given a sentence s with a pre-identified set of variables v and types t , the task is defined as a binary classification of all edges v × t , where a positive edge means that the variable is assigned that type and negative otherwise. figure 1 introduces an example of a mathematical statement, containing one variable b with type persistence length. the edge linking to this type is a positive one, while the one linking to type chains is a negative edge. while the task carries some similarity to two other well-known tasks in nlp, coreference res-the restriction on the allowed conformational space is so severe that it is able to induce a finite persistence length in the modelled chains. ## our approach: variable slots (varslot) following the recent literature @xcite , this work frames the variable typing problem as binary classification, where each variable is tested against all possible types. given a sentence s from a mathematical text containing known variables @xmath0, v 2 , . . . , v nv } and types @xmath1, t 2 , . . . , t nt }, varslot aims at finding a function such that f (v i , t j ) = 1 if v i has type t j and f (v i , t j ) = 0 otherwise. in this section we detail our approach, varslot, illustrated in figure 2 . ## experiments this section presents the experiments performed to evaluate the performance of varslot for the variable typing task. the models and datasets used are made available in our repository foot_0 . as the encoder for the model, we use the sentence transformers @xcite version of scibert @xcite pre-trained for nli tasks (scibert-nli) foot_1 . we compare our approach with previous research and standard pre-trained language models. the evaluation is conducted in two different settings: ## related work even though mathematical text is often a crucial element of scientific discourse, the area of mathematical language processing is still widely unexplored @xcite . while initial efforts have focused on rule-based and bag of words-based approaches @xcite @xcite , recent work transitioned to supervised-learning methods. @xcite presents three different models for addressing this problem, framing it as a binary classification between a variable and a type. the models proposed are an svm model using features that are type and variable-centric, a convnet using pretrained embeddings to represent the input tokens and a bidirectional lstm model that takes as input the full sentence along with the pairs for classification. @xcite shows that neural models vastly outperform models with manual feature engineering. the authors also introduce a dataset for the task. ## conclusion in this paper, we introduced varslot, a model for solving the task of variable typing generating a more generalisable representation of variables. in order to evaluate the proposed encoding, with a particular emphasis on variable semantics, we propose a new setting for the variable typing task, where during inference type, the model is only exposed to new variable names. we observed that in this setting, there is a decrease in performance for all tested models; however, varslot was the most robust model. as future work, we leave the application of these specialised embeddings for different downstream tasks, similar to the work done in @xcite . we expect that generating better representations for variables will enable an increase in performance across different mathematical language inference tasks. moritz schubotz, alexey grigorev, marcus leich, howard s cohl, norman meuschke, bela gipp, abdou s youssef, and volker markl. 2016. semantification of identifiers in mathematics for better math information retrieval. in proceedings of the 39th international acm sigir conference on research and development in information retrieval, pages 135-144. georgios spithourakis and sebastian riedel. 2018. numeracy for language models: evaluating and improving their ability to predict numbers. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2104-2115. yiannos stathopoulos, simon baker, marek rei, and simone teufel. 2018. variable typing: assigning meaning to variables in mathematical text. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 303-312. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, lukasz kaiser, and illia polosukhin. 2017. attention is all you need. in nips.