Extracting Operator Trees from Model Embeddings (2022)

Abstract: Transformer-based language models are able to capture several linguistic properties such as hierarchical structures like dependency or constituency trees. Whether similar structures for mathematics are extractable from language models has not yet been explored. This work aims to probe current state-of-the-art models for the extractability of Operator Trees from their contextualized embeddings using the structure probe designed by Hewitt and Manning. We release the code and our data set for future analysis.

## introduction transformer-based language models have not only a high impact on all domains in natural language understanding but also on related fields that besides natural language try to model artificial languages such as programming code or mathematical notation written in l a t e x @xcite . the knowledge or linguistic properties that models like bert or roberta capture have been the subject of several studies: according to recent research, bert encodes information about part-of-speech tags, roles, and syntactic features such as constituency and dependency trees @xcite . since transformerencoder-based models were applied successfully for mathematical question answering or notation prediction @xcite , these models must have also acquired mathematical knowledge. however, the field of interpretability for mathematical information has not been a topic of research so far. therefore, this work aims to analyze the prevalence of one type of mathematical knowledge: operator trees, a type of parse trees that can be generated from l a t e x formulas. generally, whether a model encodes a certain property is evaluated by applying a probe, i.e., a classifier that is trained on top of the contextualized embeddings. the performance of this classifier is used as an indicator whether the information about the property was encoded in the contextualized embeddings. to analyze whether it is possible to reconstruct an operator tree from the contextualized embeddings of a transformer-encoder model, we apply the structural probe introduced by @xcite . this probe approximates the distance between nodes in the trees using the distance of two embeddings. in total, we train the structural probe on the embeddings of each layer of nine models for math and science and show that in most cases it is possible to reconstruct operator trees from the models' contextualized embeddings. the highest correlation between the learned tree distance and the gold standard is reached in the middle layers, e.g., around layer 6 for models based on bert-base and robertabase. as hewitt et al. also found for dependency trees, most models follow a similar pattern of information spreading among layers. ## related work within the last years, several transformer-encoderbased models for mathematics have been developed with different applications in mind. the recent arqmath lab 3 @xcite included several teams that applied models pre-trained on math: mirmu used mathberta, a model based on roberta-base @xcite , @xcite adapted albertbase-v2 and roberta-base for math, and @xcite further pre-trained a bert model. in arqmath lab 1, the team psu also released a further pre-trained model based on roberta @xcite . in addition, @xcite fine-tuned a bert model for notation prediction tasks based on scientific documents. mathbert @xcite leverages operator trees during pre-training for several tasks such as formula topic classification and information retrieval. related to mathematics is also the domain of scientific documents for which scibert was trained @xcite . however, little is known so far about what bert-based models learn about mathematics. in contrast, their learning capacities on natural language received large attention in recent research (for a survey see @xcite ). several probes and classifiers were employed to analyze whether bert captures grammatical structures like dependency or constituency trees @xcite @xcite or which layer attends to which linguistic feature @xcite . visual frameworks like bertviz by @xcite support the analysis of bert's inner working by visualizing the attention weights of trained models. also albert was shown to capture part-of-speech tags in different places as reported by @xcite , but most studies were performed using bert. ## probing for mathematical structures we analyze whether it is possible to reconstruct mathematical parse trees from the models' contextualized embeddings. it was already shown that bert is able to learn grammatical structures of natural languages which could be extracted in the form of constituency and dependency trees @xcite . therefore, we apply the same type of probe to test for operator trees. ## experimental setup we evaluated in total 13 models which are publicly available on the huggingface model hub foot_2 . we chose the eight mathematical models by searching the model hub for transformer-encoder models that were (further) pre-trained on mathematics. we also added the popular model scibert as its domain, science, is close to mathematics. in addition, the four models which served as a base for pre-training were evaluated. a summary of the models can be found in tab. 1. of particular interest would have been an evaluation of mathbert by @xcite , a model that relied on operator trees during pre-training. however, neither the model nor the code are publicly available. bert, albert and roberta were trained on a general natural language corpus and serve as baselines. six models were further pre-trained from a base model like bert or roberta, while three were developed from scratch. the data sources the models were trained on are rather diverse: five models use arqmath, others use math text books, school curricula, paper abstracts, or other discussion posts apart from arqmath. scibert is the only model what was not specifically trained on mathematical content, but on scientific publications. all models can be found on huggingface by using their model identifier. ## results tab. 2 summarizes the highest values from all layers. we report our results using uuas and dspr. where higher values indicate a larger percentage of correctly reconstructed edges and a higher correlation between the predicted and gold-distances, respectively. each value is the mean of the five runs. it is visible that almost all adapted models improve over their natural language baselines. the highest performance overall is demonstrated by an-reu/math_pretrained_bert. only the performance of mathbert-custom drops in comparison to bertbase-cased. the dspr. scores of the best models in comparison to their baselines are visualized in fig. 2 . in general, the models pre-trained on arqmath demonstrated a better performance across both metrics compared to models pre-trained on other data sets. a possible reason could be that this data set contains a large variety of formulas written in l a t e x while this is unclear for the other data sets since they are not publicly available. we validated these results also using a second opt data set based on the math data set @xcite , which contains formulas written in l a t e xextracted from competition math problems. since there was no drop in performance among the models pre-trained on arqmath, we can conclude that models did not benefit from the overlap between the pre-training data and the probing formulas. bert and roberta-based models show that the best extractability for operator trees lies in the middle layers, between layer 4 and 7 for base models and between layer 9 and 13 for large models. this pattern is consistent with the results reported by hewitt et al. for dependency structures. notably, the same pattern does not emerge for albert and anreu/math_albert. here, the highest scores are in layers 2 and 3. overall, the scores for both albert-based models are significantly lower, even after training on arqmath. interestingly, this model was among the best for the arqmath lab 3 on mathematical answer retrieval and outperformed also anreu/math_pretrained_roberta, which is the second best model for uuas in this study. a similar mismatch between the perfor- ## conclusion this work aims to answer the question: are operator trees extractable from the models' contextualized embeddings? we trained a structural probe that learns to approximate the distances between nodes in the trees. the results show that models (further) pre-trained on mathematical data sets outperform their natural language baselines. the high correlation of the trained probe suggests that the models indeed encode useful information about operator trees in their contextualized embeddings. given that the models have never been trained on operator trees, but only using masked-language modeling on string-based representation such as l a t e x, explicitly proving operator trees during a downstream task such as mathematical retrieval might not even be necessary. furthermore, we notice differences between model classes: while bert and roberta-based models demonstrate a higher extractability for the structural probe, both albert-based models fall behind. in contrast, their performance on mathematical answer retrieval is on par with the other evaluated models. further research is required to investigate this issue. we are open to offer other researchers the re-use of our work by making our source code and data set fully publicly available on github 7 .