W ireless M ath B ench: A Mathematical Modeling Benchmark for LLM s in Wireless Communications (2025)

Abstract: Large Language Models (LLMs) have achieved impressive results across a broad array of tasks, yet their capacity for complex, domain-specific mathematical reasoning—particularly in wireless communications—remains underexplored. In this work, we introduce WirelessMathBench, a novel benchmark specifically designed to evaluate LLMs on mathematical modeling challenges to wireless communications engineering. Our benchmark consists of 587 meticulously curated questions sourced from 40 state-of-the-art research papers, encompassing a diverse spectrum of tasks ranging from basic multiple-choice questions to complex equation completion tasks, including both partial and full completions, all of which rigorously adhere to physical and dimensional constraints. Through extensive experimentation with leading LLMs, we observe that while many models excel in basic recall tasks, their performance degrades significantly when reconstructing partially or fully obscured equations, exposing fundamental limitations in current LLMs. Even DeepSeek-R1, the best performer on our benchmark, achieves an average accuracy of only 38.05%, with a mere 7.83% success rate in full equation completion. By publicly releasing WirelessMathBench along with the evaluation toolkit, we aim to advance the development of more robust, domain-aware LLMs for wireless system analysis and broader engineering applications.

## introduction large language models (llms) have recently demonstrated groundbreaking performance across a diverse range of natural language tasks-from general language understanding @xcite and code generation @xcite to elementary mathematical reasoning @xcite . advanced models such as @xcite and deepseek-r1 @xcite figure 1 : example task from wirelessmathbench a system model derivation from wireless communications literature. the derivation progresses from a multiplechoice question to progressive mask completion questions, and finally to the full formula derivation, testing the model's ability to reason through complex channel reflections and matrix operations. in many engineering fields-wireless communications in particular-mathematical modeling is indispensable. the design and analysis of modern wireless systems require not only accurate numerical computation but also precise symbolic derivations that honor strict physical and dimensional constraints. tasks such as channel estimation @xcite @xcite , beamforming @xcite @xcite , and multi-antenna system @xcite @xcite involve intricate matrix operations, multi-stage derivations, and domain-specific lexicon. even minor errors in symbolic manipulation can lead to significant performance degradation or non-compliance with industry standards @xcite . although recent work has leveraged llms for technical definition retrieval in wireless communications @xcite @xcite , few studies have directly addressed the challenges associated with multi-step derivations and symbolic manipulation in this specialized domain. this observation raises a broader question: to what extent are llms capable of emulating the mathematical derivations and analytical typically by an engineer or researcher in the field of real wireless communications? to bridge this gap, we introduce wirelessmath-bench a comprehensive benchmark specifically designed to test llms on the real-work wireless engineering orientation mathematical reasoning. wirelessmathbench comprises 587 high-quality questions sourced from 40 state-of-the-art papers, each carefully annotated and validated by domain experts to ensure accuracy. these questions span a variety of system models (e.g., multiple-input and multiple-output (mimo), non-orthogonal multiple access (noma), reconfigurable intelligent surfaces (ris) ) and problem settings (e.g., channel estimation, beamforming), encompassing multiplechoice, fill-in-the-blank, and open-ended questions at various levels. table1 highlights key differences between wirelessmathbench and other math benchmarks, ours is the only dataset of expert difficulty level and contains real-world engineering problems. figure 1 illustrates how a single math formula escalates from a basic multiple-choice query to a fully masked equation derivation, reflect-ing the complexity of real-world wireless system analysis. our extensive experiments show that while leading llms perform well on simpler tasks (e.g., multiple-choice questions with over 75% accuracy), their performance drops dramatically on advanced derivation tasks (progressive masking and complete equations). even the strongest model we evaluated, deepseek-r1 @xcite , only manages a 7.83% success rate in fully masked derivations, underscoring a fundamental gap between current llm capabilities and the complex demands of wireless systems analysis. by publicly releasing wirelessmathbench along with its evaluation toolkit, we aim to spur progress toward llms that are not only fluent in natural language but also capable of rigorous, domainspecific mathematical reasoning. we envision that wirelessmathbench will serve as a catalyst for innovation in mathematical reasoning capabilities, domain-adaptive pre-training techniques, and advanced thought-chaining strategies, ultimately propelling llms toward more robust scientific and engineering problem-solving. ## the wirelessmathbench benchmark in this section, we present wirelessmathbench, a new benchmark specifically designed to evaluate llms on mathematical modeling tasks within wireless communications. we begin by discussing the rationale behind our benchmark design (section 3.1), followed by the details of our data collection and annotation pipeline (section 3.2). we then explain how we construct questions of varying complexity levels, as well as our progressive masking methodology (section 3.3). ## experiments we now present a comprehensive evaluation of wirelessmathbench, focusing on how leading llms handle wireless communications-specific mathematical modeling. we begin by detailing our experimental setup (section 4.1), including prompt design and model baselines, then discuss our main results (section 4.2), and conclude with an error analysis that highlights key challenges (section 4.3). ## conclusion we introduced wirelessmathbench, the first benchmark that comprehensively evaluates llms' abilities to domain-specific mathematical engineering tasks in wireless communications. by presenting a broad range of tasks-from multiplechoice questions to progressively masked fillin-the-blank and full equation completions-wirelessmathbench assesses how well models handle real-world wireless communications mathematical engineering challenges. our experiments show that, although many leading llms perform well on simpler question types, their ability to reconstruct equations deteriorates significantly when partial or full derivation is required, exposing a critical shortfall in current llm-assisted scientific innovation. future work will expand the scope and complexity of these wireless challenges, with the aim of enhancing llms' mathematical reasoning and domain adaptation. by advancing their integration into the next-generation wireless systems, we ultimately strive toward the development of more capable, general-purpose ai solutions for scientific and engineering applications.