Probing for Multilingual Numerical Understanding in Transformer-Based Language Models (2020)

Abstract: Natural language numbers are an example of compositional structures, where larger numbers are composed of operations on smaller numbers. Given that compositional reasoning is a key to natural language understanding, we propose novel multilingual probing tasks tested on DistilBERT, XLM, and BERT to investigate for evidence of compositional reasoning over numerical data in various natural language number systems. By using both grammaticality judgment and value comparison classification tasks in English, Japanese, Danish, and French, we find evidence that the information encoded in these pretrained models’ embeddings is sufficient for grammaticality judgments but generally not for value comparisons. We analyze possible reasons for this and discuss how our tasks could be extended in further studies.

## introduction in recent years, transformer-based language models such as bert @xcite , xlm @xcite , and distilbert @xcite have achieved unprecedented results on a wide array of natural language understanding tasks, even when such models are trained on other tasks (i.e. transfer learning). in light of this success, there has been increased interest in investigating what particular information transformer-based language models encode in their word embeddings during pretraining that allows them to perform well in transfer learning experiments. put into the context of this paper, we may ask: do such models gain certain linguistic/compositional understanding from pretraining? attempts to assess such phenomena are commonly referred to as probing experiments. in this paper, we introduce a novel probing task using targeted datasets and classification tasks aimed at evaluating models' compositional reasoning capabilities in relation to numbers 1 in multiple natural languages. we choose both grammaticality judgment and value comparison tasks (section 3) to assess multilingual distilbert, xlm, and bert 2 over various number systems. we argue that high performance on these tasks indicates some ability of reasoning over compositional structures, particularly over the rules generating valid compositional structures (task 1) and the resultant meanings of the structures (task 2). after probing the selected models on our tasks, we discuss explanations for the performance of models, as well as possible future extensions to this probing task schema. additionally, studies in cognitive psychology such as @xcite assert that children learning more transparent number systems (i.e. those exhibiting more regularity in their surface forms such as japanese) have a greater counting proficiency in several tasks compared to those learning less transparent (opaque) systems, such as english or french. although it is not the focus of our work, given the multilingual setting, we will refer to the idea of number system transparency when analyzing possible explanations of results. 3 2 related work our approach is informed by previous linguistically-motivated probing studies such 1 for our purposes, numbers are spelled out, i.e. written out as words such as "ninety". at those discussed in @xcite and @xcite . though ettinger (2020) discusses important findings on psycholinguistic probing experiments of bert, we find @xcite particularly useful for our study due to its clear explanations of linguistic probing experiment setup. in their study, ettinger et al. present methods for constructing linguisticallytargeted datasets (including example sentences, as we use) and classification tasks for probing word embeddings for semantic knowledge. as one of our tasks also seeks to probe for semantic knowledge, we were able to use this setup as a rough guideline. in addition, the authors are careful to create linguistic data with sufficient diversity as not to give potentially helpful cues to their classifier which are not related to the knowledge they wish to probe for. we thus carefully create our task data in a similar manner by limiting the distribution of our data (described more later) and forbidding duplicates. to our knowledge, there have been few studies conducted on investigating numerical understanding specifically in transformer-based language models with a multilingual approach. however, one particularly relevant study on english comes from @xcite . wallace et al. probe the embeddings of various models (bert, elmo, word2vec, etc.) using three tasks: find the maximum of a list, decode a number word to its numerical form, and add number words to produce a numerical form. in their results, the authors note that all embeddings contain some understanding, though standard embeddings perform particularly well and character-level models perform best. although we investigate similar phenomena as wallace et al., our methodology includes several key differences: in this way, we hope to make our tasks/data as restrictive as possible in order that they require a certain compositional/linguistic understanding. ## methods we propose and perform two classification tasks in english, danish, japanese, and french. task 1 is a probe for underlying syntactic information encoded in pretrained word embeddings, while task 2 is a probe of underlying semantic information. we run our tasks on all three models over two different datasets which we have generated: one where number words are inserted into sentences (e.g. "there are seven hundred books in the library.") and one with numbers alone (e.g. "seven-hundred"). our probing model features a multilayer perceptron (a nn with a single hidden layer) classifier on top of the existing transformer language model architecture. in this manner, pretrained word embeddings from the language model (bert, distilbert, xlm) are fed as input to the mlp classifier which itself is then trained on our tasks. a depiction of the probing model structure is shown in figure 1 . ## results the following sections show probing results on both tasks. before probing and as a precaution, we fine-tuned our models on both tasks. as we expected, we find that both tasks are learnable to accuracies above 95%. a further discussion of finetuning results is left for appendix a. ## further work as this work is an exploration of a new probing method for state-of-the-art language model architecture, there are surely a number of ways to extend from it. though we discussed it briefly here, exploring the architectural reasons for the shortcomings of these pretrained embeddings, especially in the case of task 2 and with sentences is an important area for future work. indeed, in a similar task from @xcite , bert was also found to have poor performance. in the future, several more specific probes could be designed to test for understanding of magnitude in various linguistic contexts to find strengths and weaknesses of transformer-based models. a particularly interesting case would be in testing magnitude comprehension in sentences of varying structures. our sentence templates used in this study are few, and experimenting with other varieties could prove to be insightful. our experiment also made use of the idea of language transparency. we also find this to be a topic for possible further work. namely, is there a method to reliably measure transparency of languages to predict performance on numerical understanding tasks such as these? we believe this may be possible through measuring complexities of grammars which generate number words in each language. overall, in future extensions of this study, there is room for more languages, sentence types, task renditions, and models. ## conclusion in this paper, we introduced methods for probing the multilingual compositional reasoning capabilities of transformer-based models' pretrained embeddings over natural language numbers. from our experiments, we've shown that these pretrained embeddings show some capabilities in making grammatical judgments of number words, though they are less capable of making value com-parisons. in addition, we find that results generally follow a trend based upon model size. our results are in accord with previous work such as @xcite ; however, we have also highlighted further model weaknesses through our probing methods. therefore, the opportunities for future work, especially with a multilingual focus, are plenty. num2words (ogawa) package. for the ungrammatical data, two grammatical numbers are randomly generated, both converted to text, then appended together to create an ungrammatical number. for example the ungrammatical number "fifty-five two hundred" is the combination of "fifty-five" and "two hundred". another example made from the same original elements could be: "two fifty-five hundred". grammatical numbers used in splits, however, were only split such that the resulting elements were grammatical words themselves. so, for example, an non-continuous number word string like "fi-tfy te nnine" would never occur. since generating numbers with appendage can naturally occur in ungrammatical number words being longer than grammatical, we control for length by limiting our set of ungrammatical number words to words that are at most as long as the longest grammatical number. lastly, we ensure no grammatical numbers are accidentally created in this process by keeping a list of known grammatical numbers in text form (generated by num2words) that ranges from number sufficiently higher than our generation range. for example, if our number word generation ranges from 0-1000, we would make this list of known grammatical numbers from 1-100,000,000. these number words are finally either left bare or inserted into sentences to form our x inputs and are labeled 0 if grammatical and 1 if ungrammatical.