NUMC o T : Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models (2024)

Abstract: Numeral systems and units of measurement are two conjoined topics in activities of human beings and have mutual effects with the languages expressing them. Currently, the evaluation of Large Language Models (LLMs) often involves mathematical reasoning, yet little attention is given to how minor changes in numbers or units can drastically alter the complexity of problems and the performance of LLMs. In this paper, we scrutinize existing LLMs on processing of numerals and units of measurement by constructing datasets with perturbations. We first anatomize the reasoning of math word problems to different sub-procedures like numeral conversions from language to numbers and measurement conversions based on units. Then we further annotate math word problems from ancient Chinese arithmetic works which are challenging in numerals and units of measurement. Experiments on perturbed datasets demonstrate that LLMs still encounter difficulties in handling numeral and measurement conversions.

## introduction numbers and counting are the basic concepts in human experience. numbers are a set of conceptual tools made from words and other symbols for specific quantities and a key set of linguistically based innovations that distinguish human species from others @xcite . the development of numeral systems allows humans to express numbers in a consistent manner. 1 counting is usually not a monotone process of manipulating numbers from a numeral system but to quantify objects with a units of measurement foot_1 to compare the magnitude. in the literature, @xcite adopt the taxonomy discipline called core systems of number @xcite ) from cognitive science. the tasks in numeracy are then categorized by the granularity and units attached to the quantities in the task, where granularity means whether the encoding of the number is exact or approximate, and units represent whether the numerals are in their numerical forms or grounded with units of measurement. based on the taxonomy, existing numeracyoriented tasks are identified as simple arithmetic tasks @xcite , numeration tasks @xcite @xcite , magnitude comparison tasks @xcite , math word problems (mwps) @xcite @xcite , exact facts in the context of numeracy @xcite , measurement estimation tasks @xcite @xcite and numerical language modeling tasks. there are still tasks which fall out the taxonomy, such as numeric paraphrasing (one-to-one correspondences between different surface forms of the same number), quantity entailment tasks @xcite , numeral understanding tasks, fused-head resolution, counting tasks @xcite and other domain-specific tasks. as far as we are concerned, the tasks discussed above cover a wide range of topics in numeracy and address a lot of challenges faced by numerals and units of measurement. however, we still need to address the issue of numeracy when discussing arithmetic by using pure numerals and making an extra effort to take units of measurement into consideration. the inadequacy in accurately converting numerals with units of measurement may lead to unpredictable consequences in real-life scenarios, especially in the era of large language models (llms) where decoderonly generation methods are being employed. ## experiments we conduct the experiments using open-sourced llms as well as api-based llms supporting both english and chinese languages. for publicly available llms, we chose chatglm2-6b 3 and three models from the llama-2 4 family: 7b, 13b, and 70b, which were deployed to a6000 gpu server locally. for api-based llms, we use chatgpt 5 , ernie-bot-turbo 6 , chatglm-turbo 7 . we consider the following prompt: (1) zero-shot：we simply present the questions to the ## conclusion we investigate the performance of various llms on tasks involving numeral conversions and units of measurement in both chinese and english languages. additionally, we explore the capability boundaries of llms by introducing cot and external knowledge. based on a series of experiments, the conclusions are as follows: