Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models (2025)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in natural language processing tasks, such as text generation and semantic understanding. However, their performance on numerical reasoning tasks, such as basic arithmetic, numerical retrieval, and magnitude comparison, remains surprisingly poor. This gap arises from their reliance on surface-level statistical patterns rather than understanding numbers as continuous magnitudes. Existing benchmarks primarily focus on either linguistic competence or structured mathematical problem-solving, neglecting fundamental numerical reasoning required in real-world scenarios. To bridge this gap, we propose NumericBench, a comprehensive benchmark to evaluate six fundamental numerical capabilities: number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. NumericBench includes datasets ranging from synthetic number lists to crawled real-world data, addressing challenges like long contexts, noise, and multi-step reasoning. Extensive experiments on state-of-the-art LLMs, including GPT-4 and DeepSeek, reveal persistent weaknesses in numerical reasoning, highlighting the urgent need to improve numerically-aware language modeling. The benchmark is released in: https://github.com/TreeAI-Lab/NumericBench.

## introduction large language models (llms) @xcite have demonstrated remarkable capabilities in text generation, semantic understanding, and task adaptation across diverse domains @xcite . their success is largely attributed to pretraining on vast text corpora using next-token prediction objectives @xcite , which enables generalization to tasks requiring linguistic creativity, commonsense reasoning, and domain-specific knowledge @xcite . however, while llms perform well in text understanding tasks and olympic mathematics questions @xcite , they surprisingly struggle with simple numerical tasks such as basic multiplication, comparison, and retrieval. as shown in figure 1 , effective gpt-4o fails at simple numerical tasks, even with number comparisons. unlike tasks that rely primarily on semantic coherence and linguistic structures, numerical reasoning requires a deeper understanding of numbers as continuous magnitudes rather than discrete tokens. current llms tend to prioritize surface-level statistical patterns, such as lexical co-occurrences and syntactic regularities, which limits their ability to process numerical operations @xcite @xcite . as a result, llms frequently struggle with tasks involving numeric retrieval, arithmetic operations, and magnitude comparisons. these shortcomings highlight an urgent need to systematically evaluate and improve the numerical reasoning capabilities of llms. current evaluation frameworks for llms prioritize either linguistic competence or formal mathematical problem-solving. for instance, semanticoriented benchmarks @xcite , such as glue @xcite , superglue @xcite , and simpleqa @xcite , primarily assess linguistic competence and semantic understanding, while math-oriented benchmarks, such as mathqa @xcite , gsm8k @xcite , and mathbench @xcite , focus on structured algebraic or geometric tasks. however, these approaches neglect the basic demands of real-world numerical reasoning applications and fundamental numerical abilities, where numbers are often embedded in unstructured and noisy context data. for example, analyzing fluctuating stock prices or weather requires basic numeric retrieval, comparison, and summery abilities. to address the limitations of existing benchmarks, we propose a comprehensive benchmark numericbench, which consists of six general datasets, i.e., arithmetic numbers, mixed-numberstrings, number lists, stock, weather, and numerical sequences with patterns. unlike prior benchmarks, numericbench systematically evaluate the six fundamental numerical abilities of llms: (1) number recognition: it evaluates the ability of llms to identify numbers within dense strings. by integrating six datasets, ranging from synthetic numerical lists to crawled real-world data, numericbench evaluates six fundamental numerical abilities of llms across various scenarios, such as noisy contexts. our experimental analysis over various effective series of llms, including gpt-4 @xcite , deepseek @xcite , and llama @xcite , reveals persistent weaknesses in handling these fundamental numerical tasks. we further analyze five potential reasons behind these numerical reasoning gaps, including tokenizer, training corpora, training paradigms, positional embeddings, and architectural constraints. these findings offer actionable insights to guide future numerical ability improvements for llms. given that numerical reasoning is critical for real-world problem-solving, it represents a cornerstone for the development of artificial general intelligence (agi). this emphasizes the urgent need to advance numerically-aware language modeling. the contributes of this paper are summarized as follows: numericbench evaluates six fundamental numerical reasoning abilities, including number recognition, arithmetic operations, contextual retrieval, comparison, summary, and logic reasoning. ## preliminary and related works in this section, we first introduce large language models and then present existing benchmarks. ## numericbench in this section, we present our created numer-icbench, which is specifically designed to evaluate fundamental numerical capabilities of llms. nu-mericbench consists of diverse datasets and tasks, enabling a systematic and comprehensive evaluation. we discuss the datasets included in nu-mericbench, the key abilities it evaluates, and the methodology for benchmark generation. ## conclusion and future directions in this paper, we propose a comprehensive benchmark numericbench to evaluate the six fundamental numerical abilities of llms, including number recognition, arithmetic operations, contextual retrieval, comparison, summary, and logical reasoning. our experiments reveal significant gaps in llms' numerical reasoning, as even state-of-theart models like gpt-4o and deepseek-v3 struggle with simple arithmetic, number retrieval, and multistep reasoning tasks. these shortcomings arise from tokenization issues, training paradigms, and architectural limitations, underscoring the need for more numerically-aware modeling approaches. to address these gaps, several future directions deserve exploration. first, developing numericallyaware tokenizers that treat numbers as continuous magnitudes can enable llms to better understand numerical concepts. second, designing pretraining objectives specifically tailored to numerical reasoning, rather than relying solely on next-token prediction, can help models become more proficient at solving numerical problems. third, incorporating structured numerical datasets during training can enhance real-world applicability by grounding models in accurate and practical numerical contexts. finally, exploring suitable positional embeddings and hybrid symbolic-numeric architectures shows significant promise for improving the numerical capabilities of llms.