Learning Mathematical Properties of Integers (2021)

Abstract: Embedding words in high-dimensional vector spaces has proven valuable in many natural language applications. In this work, we investigate whether similarly-trained embeddings of integers can capture concepts that are useful for mathematical applications. We probe the integer embeddings for mathematical knowledge, apply them to a set of numerical reasoning tasks, and show that by learning the representations from mathematical sequence data, we can substantially improve over number embeddings learned from English text corpora.

## introduction word vector representations learned by neural models have been shown to capture linguistic knowledge that can be useful for downstream nlp tasks @xcite . in this work, we look at whether similarly-trained integer embeddings can represent useful mathematical knowledge. as a simple example, a mathematician may look at a few values of the function 2 n -1: n 1 2 3 4 5 6 7 ... 2 n -1 1 3 7 15 31 63 127 ... and notice patterns such as 'if n is even, then 2 n -1 is divisible by 3'. this type of reasoning is frequent in the early stages of mathematical work. in order for software to assist people in identifying such patterns, it needs to have an internal representation of the basic properties of integers. just as learned word representations can capture attributes like the word's part of speech or syntactic dependency label @xcite , we would like to develop integer embeddings that capture primality, divisibility by 3, etc. in this paper, we learn integer embeddings from mathematical resources and probe them for mathematical knowledge. unlike most natural-language features, many number-theoretic properties are highly regular and often may be easily determined from the integer value itself (e.g. converting an integer to its binary representation instantly reveals whether it is a power of 2). however, our focus is not on building classifiers to predict these properties, but rather on exploring whether this information can be learned from the integer co-occurrence statistics in mathematical data and how it is distributed across the vector's dimensions. besides that, we are also interested in whether the integer representations learned from context can capture meaningful mathematical properties without us having to define them explicitly. it is not clear how one would probe for such unspecified features, but we can instead estimate the usefulness of the learned embeddings for downstream numerical reasoning tasks. in this work, we apply several types of integer embeddings to a set of tasks built around mathematical regularities, and show that embeddings learned from integer sequences yield better performance than the ones trained on text. the sequential structure of oeis allows us to use it for training nlp models developed for textual data, such as recurrent neural network (rnn) language models and co-occurrence-based word embeddings. of the 336k sequences in oeis, we allocate 90% for training and divide the rest into development (used for hyperparameter tuning) and test splits; split statistics are listed in ## integer embeddings in our experiments, we compare the integer embeddings learned from oeis with the representations of integer tokens in the vocabulary of word embedding models trained on english text. ## what's in an integer embedding? for the learned vectors to be useful for mathematical applications, they need to contain the information about the important integer properties, and this information can be distributed over one or several neurons. for example, in the lstm embeddings trained on oeis, we quickly find an "evenness neuron": the 156th element of the embedding vector v is generally positive for even numbers and negative for odd numbers. this holds for integers from 1 to 50 with only a few exceptions, e.g.: probing classifiers. we use probing classifiers to test whether the embeddings learn mathematical properties. to avoid spurious correlations, we keep our classifiers very simple. we start by probing for three binary properties: divisibility by 2, 3, and 4, and primality. we train a logistic regression classifier on integers from 1 to 1000 and evaluate its predictions on integers from 1001 to 2000. all analogy problems are of the form a:b :: c:?, and the task is to choose one of the 3-5 provided options that would most appropriately fill in the blank. the column labeled "?" shows the expected correct answer, and the numbers shaded in grey are the remaining answer options. in our experiments with integer embeddings, we use vector arithmetic to find the projected vector of the answer and then select its nearest neighbor among the presented options. ## conclusion we introduce integer embeddings trained on the online encyclopedia of integer sequences (oeis) and probe them for mathematical knowledge along with the integer representations found in the vocabulary of pre-trained english word embeddings. we find the oeis embeddings promising for mathematical applications, as they are able to capture some meaningful mathematical regularities.