N umero L ogic: Number Encoding for Enhanced LLM s’ Numerical Reasoning (2024)

Abstract: Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before each number. For instance, instead of “42”, we suggest using “2:42” as the new format. This approach, which we term NumeroLogic, offers an added advantage in number generation by serving as a Chain of Thought (CoT). By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number. We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic formatting. We further demonstrate NumeroLogic applicability to general natural language modeling, improving language understanding performance in the MMLU benchmark.

## introduction large language models (llms) struggle with numerical and arithmetical tasks. despite continuous improvements, even the most advanced models like gpt-4 @xcite still exhibit poor performance when confronted with tasks such as multiplying 3-digit numbers @xcite . recent studies @xcite ) have proposed techniques to improve arithmetic in llms, such as the chain of thought (cot; @xcite ) method, which pushes the model to anticipate the entire sequence of algorithmic steps rather than just the final output. while these strategies offer valuable insights into the capabilities of llms, they primarily concentrate on post-hoc solutions for specific arithmetic challenges and do not present a practical solution for pretraining llms. the model has to reach the final digits of a number before it can infer the place value of the first digit. to address this, we propose "numerologic", a numerical format where digit count is indicated before the actual number. image by dall-e 3 @xcite . our research, however, focuses on solutions applicable to self-supervised language modeling in general, utilizing arithmetic exercises primarily for evaluating their impact. we hypothesize that one of the challenges llms face when dealing with numerical tasks is the textual representation of numbers. in today's most popular decoder-based llms, each token attends only to previous tokens. when a model "reads" a token representing a digit (or multiple digits) it cannot tell its place value, i.e. '1' can represent 1 million, 1 thousand, or a single unit. only when reaching the end of the number might the model update its representation of the previous digit tokens to be related to their real place value. to address this issue, we propose a straightforward reformatting technique called "numero-logic," which involves adding the number of digits 206 as a prefix to numbers. this lets the model know in advance what is the place value of a digit before it is read. this simple change also offers another benefit, when the model is generating a number it needs to first reason about what is going to be the number of digits. this acts as a chain of thought (cot) @xcite , encouraging the model to perform some reasoning before it begins to predict digits. implementing the suggested reformatting does not necessitate any alterations to the model's architecture; it can be accomplished through text pre-and post-processing based on regex. we demonstrate that numerologic enhances the numerical abilities of llms across both small and larger models (up to 7b parameters). this enhancement is showcased through supervised training on arithmetic tasks and its application in selfsupervised causal language modeling to enhance general language comprehension. ## related work recently, there has been a significant interest in enhancing the numerical capabilities of llms. one approach to investigating these capabilities is by assessing their performance in arithmetic tasks. several recent studies have proposed methods to enhance performance in these tasks. one strategy involves reversing the expected result order from the least to the most significant digit @xcite . another strategy is using an elaborated cot where the model is taught to predict all steps of an algorithm predefined for each arithmetic task @xcite . in @xcite , it is noted that the model learns to rely too heavily on positional encoding when trained for a specific arithmetic task. they suggest ways to overcome it, e.g. adding random white spaces in the middle of numbers. these studies aim to enhance the performance of arithmetic tasks by offering tailored solutions to the associated challenges. in contrast, our focus is on identifying solutions that benefit general language modeling rather than just arithmetic tasks, with arithmetic tasks being used solely for measuring improvements. another aspect important for llms numerical capabilities is the tokenization process. the commonly used byte pair encoding (bpe) based methods @xcite for tokenization are based on the corpus distribution and can split a number to tokens in unintuitive ways. different foundation models took different approaches when dealing with number tokeniza-tion. palm @xcite , llama @xcite force each digit to have a single token. gpt-3.5 and gpt-4 define a token for each up to 3-digit number @xcite . somewhat related to our work, in (singh and strouse, 2024), they highlighted an issue with the gpt approach. they show that dividing large numbers into 3-digit segments from left to right undermines arithmetic performance. they suggest overcoming it by inserting commas between digits to control the splitting. another related work, is @xcite . they focus on the extrapolation ability of llms to unseen numbers and use a special number encoding that lets the llm know the digit place-value. ## numerologic we introduce numerologic, a technique for boosting causal llm's numerical capabilities. the concept involves adding a digit count before numbers, enabling the model to know the place values of digits before reaching the final digits of a number. additionally, the model needs to predict the total number of digits before generating a number, acting as a simplified cot, prompting it to reason about the number that is going to be generated. we add special tokens to help represent numbers with the number-of-digit prefix, "<@xmath0, "<@xmath1, and "<@xmath2, for simplicity, "<@xmath3, "<@xmath4, and "<@xmath5. for floating points, the prefix includes both the number of digits of the integer part and the decimal part. for example, "42" is replaced by "<@xmath6.14" is replaced by "<@xmath7.2<@xmath8.14<@xmath9. when using the llm to generate numbers, we disregard the information about the number of digits and only retain the generated number itself. although not within the scope of this study, it may be feasible to leverage the additional information to identify discrepancies, wherein the model predicts a certain digit count but produces a number with a different count of digits. for small transformers, we train all parameters from scratch with character-level tokenization. for small transformers, we also replace the special tokens with single characters, "<@xmath10, "<@xmath11, and "<@xmath12, ":", and "}", respectively. for larger transformers, we start from pre-trained models. we add the new special tokens to the tokenizer's vocabulary and expand the embedding layer and the final fully connected layer to fit the new vocabulary size. when continuing training on causal language modeling or fine-tuning on supervised arithmetic tasks, we use low-rank adaptation (lora) @xcite . we apply lora for the attention block projection matrices (q, k, v, o) and train the modified embedding layer and the final fully-connected layer in full rank. the numerologic approach includes basic text pre-processing and post-processing steps that occur before and after the tokenizer's encoding and decoding methods, respectively. both can be implemented based on regular expressions: ## experiments to test the effect of numerologic we conducted several experiments. first, we tested supervised training of a small language model (nanogpt) on various arithmetic tasks. we then test the scalability to larger models (llama2-7b). finally, we test self-supervised pretraining of llama2-7b, with the suggested formatting, and test on general language understanding tasks. ## conclusions we introduced numerologic, a novel method to improve language models' handling of numerical data. our approach prefixes numbers with their digit count, enhancing models' understanding of place value and prompting better reasoning about numbers' magnitude, akin to chain-of-thought reasoning. we tested numerologic on both arithmetic and broader language understanding tasks. the results showed substantial enhancements in numerical tasks, including integer and floating-point calculations, and in broader modeling contexts like the mmlu benchmark. in summary, numerologic is a straightforward yet effective enhancement for language models' numerical abilities, applicable across various tasks without requiring changes to the models' architecture.