Do Language Models Understand Measurements? (2022)

Abstract: Recent success of pre-trained language models (PLMs) has stimulated interest in their ability to understand and work with numbers. Yet, the numerical reasoning over measurements has not been formally studied despite their importance. In this study, we show that PLMs lack the capability required for reasoning over measurements. Furthermore, we find that a language model trained on a measurement-rich corpus shows better performance on understanding measurements. We propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in the probing tasks.

## introduction the success of pre-trained language models (plms) has led to more research on their ability to understand commonsense. in this context, numerical reasoning over text (nrot) is a nlp model's ability to interpret and work with numbers in either digit or word form @xcite . recent studies on nrot test plms to answer questions on numeracy @xcite , scalar magnitude comparison @xcite , numerical facts @xcite , and math word problems @xcite . despite these efforts, existing works lack an analysis of the forms in which numbers appear. in particular, we focus on the case where numbers appear as a measurement in the context. in most scientific articles, measurements are an integral part of the context for capturing its appropriate meaning. for example, the two sentences "40g of aspirin is lethal" and "40mg of aspirin is lethal" contain the same words except for the unit of measurement (uom), but the second sentence is incorrect because of the uom. in this work, we examine the measuring skill of plms: the ability to understand the system of measurement and perform numerical reasoning over measurements. we design three measuring skill tests (msts) and study how many measuring skills can be acquired. specifically, unit conversion, reference range detection, and measure-ment comparison require understanding of the system of measurement, the normal range of the biomedical entity, and the ability to combine knowledge about the system of measurement and nrot, respectively. mst results showed that the models struggled to find the largest (or smallest) value on the list of measurements and convert the measurement to another unit, while they performed well on other tests. compared to other plms, biobert @xcite showed superior performance on unit con-version and reference range detection, which implies that pre-training with measurementrich text helps the model understand the system of measurement. finally, we speculate that the lack of skills to distinguish numbers, units, and other words in the context makes the models fail in some msts. to mitigate this, we introduce scale embedding, which provides the model with the information regarding the position and scale of the numbers in the input text. we show that scale embedding significantly improves the mst performance of all plms. ## measuring skill test in this section, we describe three msts to carefully study the ability of plms to understand the system of measurement and perform numerical reasoning over the measurements. ## experiments probing setup we formulated msts as a cloze test @xcite to fully utilize the knowledge captured by masked language modeling (mlm). specifically, a plm received the masked inputs given in we probed four transformer-based plms. bert @xcite and albert @xcite were trained on wikipedia articles and book corpus. biobert @xcite ) was trained on biomedical articles from pubmed abstracts, and bluebert @xcite used both clinical (mimic-iii @xcite ) and biomedical (pubmed abstracts) corpus for pretraining. we also tested a randomly initialized transformer encoder (i.e. scratch) to evaluate the difficulty of our msts. for each model, we did not update the parameters during training, except for the mlm head in the last transformer layer. in all tasks, the models were trained with three random seeds and we report the mean classification accuracy for all the probing tasks. appendix a provides further details on training and evaluation. data preparation we manually crafted templates in the numbers in the training dataset were sampled from [10 -2 , 10 2 ). for evaluation, we constructed two evaluation datasets: 1) interpolation sampled numbers from the same range as the training dataset; 2) extrapolation sampled numbers from [10 -3 , 10 3 ). note that we did not consider the numbers outside the range [10 -3 , 10 3 ), because many of the unit prefixes are in the power of thousands. @xcite reported that representing numbers in scientific notation made it easier for the language model to capture the scale of numbers. following this observation, we tested two different number notations: decimal and scientific. for example, 32.6 can be represented as 32.6 and 3.26e+01 in decimal and scientific notation, respectively. we randomly varied the number of digits after the decimal point between zero and three, and the significant digits were maintained after converting the number notation. for reference range detection, we collected biomedical entities from six tables in mimic-iii (input, output, lab, prescrip-tion, procedure, and chart) and chose the subset. we report the number of samples and the distribution of labels for each mst in ## results and analysis measuring skills of plms ## conclusion to the best of our knowledge, our study is the first to investigate reasoning over measurements. our analysis shows that plms lack the capability required for reasoning over measurements. we proposed a scale embedding approach that provides information on the position and scale of numbers, and it significantly increases the mst performance.